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Executive  Summary 

The  overall  goal  of  the  present  contract  was  to  provide  a  review  and  analysis  of  the  Engel  and 
Townsend  reports  “ Guidelines  for  Human  Engineering  Requirements  for  Canadian  Forces  Command 
and  Control  Information  Systems ”  and  “User  Manual:  Guidelines  for  Human  Factors  Engineering 
Requirements  for  Canadian  Forces  Command  and  Control  Information  Systems .” 

The  following  were  the  major  objectives  of  the  work. 

•  Determine  the  utility  of  the  first  document  in  assisting  the  development  by  HumaiLsyste/Ms 
Incorporated  (HSI)  of  a  draft  Human  Engineering  Program  Plan  (HEPP)  for  the  Tactical  Battlefield 
Command  System  (TBCS)  that  was  currently  under  development  by  DLR4-5. 

•  Provide  a  commentary  and  analysis  on  the  general  utility  of  the  two  documents  for  future  use  by  the 
Canadian  military  community  to  provide  guidelines  for  the  integration  of  Human  Factors 
Engineering  (HFE)  into  future  military  acquisitions  for  Command  and  Control  Information  Systems 
(CCIS). 

•  Review  how  the  presence  of  the  reports,  had  they  been  available  at  the  time  of  project  initiation, 
might  have  influenced  the  development  of  a  sample  of  military  CCIS  acquisition  projects. 

•  Comment  on  how  the  reports  might  influence  the  future  direction  of  the  TBCS  program. 

The  objectives  were  accomplished  through  a  combination  of  review  and  analysis  of  the  relevant 
material  by  HSI,  integrated  with  the  results  from  interviews  with  four  SMEs,  chosen  for  their  recent 
experiences  as  senior  managers  in  DND  CCIS  acquisition  projects. 

The  first  objective  could  not  be  strictly  accomplished  since  the  two  reports  arrived  at  some  time  after 
the  HSI  draft  HEPP  deliverable  was  due.  Nevertheless,  a  retrospective  review  was  conducted  to 
determine  what  use  the  document  might  have  been  had  it  been  available  to  HSI.  This  review  showed 
that  the  reports  contained  many  of  the  key  concepts  that  would  have  validated  HSI’s  existing 
understanding  and  knowledge  base  that  were  used  in  the  preparation  of  the  HEPP.  In  particular,  the 
reports  were  organised  in  a  manner  that  made  the  information  easy  to  extract  and  provided  an 
overarching  framework  that  was  well  integrated  and  logically  consistent.  The  reports  could  not  have 
provided  assistance  for  the  analysis  of  the  risk  and  benefits  of  the  major  HFE  activities  associated  with 
the  HEPP,  a  task  requested  by  the  TBCS  PMO. 

With  respect  to  the  second  objective,  the  overall  conclusion  was  that  the  reports  would  be  highly 
beneficial  to  the  military  community  for  providing  guidance  on  integrating  HFE  into  future  CCIS 
acquisitions  project.  A  number  of  suggestions  are  made  that  would  enhance  the  scope,  utility  and 
usability  of  the  reports.  These  issues  were  identified  both  by  HSI  and  military  SMEs  who  were 
interviewed  in  the  course  of  the  contract;  they  include  scope,  utility,  usability  and  the  relationship  with 
other  relevant  documents.  It  should  be  recognised  that  in  many  cases  these  enhancements  go  beyond 
the  scope  of  the  original  Engel  and  Townsend  work  commissioned  by  DCIEM 

Suggestions  for  broadening  the  scope  include. 

•  Widening  the  scope  of  task  and  mission  analyses  to  include  additional  core  concepts  derived  from 
NATO  STANAG  3994  and  other  relevant  documents  (also  to  include  special  analyses  for  situation 
awareness  and  communication) 

•  Integrating  and  expanding  issues  relating  to  measures  of  performance  and  effectiveness  for  CCIS 
systems 

•  Expanding  the  section  on  HF  analyses  to  incorporate  unique  HF  issues  related  to  teams  (in 
particular,  function  allocation  and  team  decision  making). 
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The  following  recommendations  are  made  as  having  potential  to  enhance  the  Utility  of  the  reports. 

•  Provide  sections  that  outline  the  “business  case”  for  the  human  factors  engineering  effort  and  to 
show  the  costs  and  benefits  of  the  major  HFE  activities. 

•  Clarify  the  relationship  between  the  HEPP  and  the  overall  systems  engineering  plan.  Also  clearly 
differentiate  the  HF  responsibilities  and  activities  done  by  the  PM  team  prior  to  contract  award  and 
throughout  the  system  development  process,  from  those  performed  by  the  contractor  and  the  IV&V 
team. 

•  Provide  more  specific  guidance  on  how  the  data  that  are  output  from  HF  analyses  can  be  used  to 
inform  the  process  of  requirements  development,  and  later,  the  detailed  system  design,  and  the  role 
of  HFE  SMEs  in  this  process. 

•  Broaden  the  scope  to  include  issues  that  may  be  unique  to  the  navy  and  air  force  communities. 

Several  suggestions  were  made  to  enhance  the  usability  of  the  reports  to  make  the  information  easier  to 
access  for  busy  military  officers.  Suggestions  included  reducing  the  surface  density  of  the  text  by 
improved  formatting,  wider  use  of  graphics,  hypertext  links  and,  most  importantly,  widespread 
illustration  of  issues  through  specific  military  examples  and  case  studies. 

A  final  suggestion  was  to  integrate  the  report  with  other  highly  relevant  material  contained  in  other 
DCIEM  technical  reports  and  NATO  documents. 

With  respect  to  the  third  objective,  there  was  good  consensus  among  the  SMEs  that  the  availability  of 
the  reports  at  the  start  of  their  respective  projects  would  have  had  a  major  beneficial  effect.  These 
benefits  would  have  occurred  in  areas  such  as  clearly  defining  existing  system  deficiencies  in  a 
disciplined  manner,  identifying  the  major  HFE  tasks  and  responsibilities,  doing  early  mission,  function 
and  task  analysis  and  providing  the  basis  for  ensuring  that  an  appropriate  HEPP  could  be  put  into  place. 

The  final  objective,  to  determine  how  the  reports  might  inform  the  future  direction  of  TBCS  was 
somewhat  moot,  given  that  by  the  time  the  present  review  was  conducted,  the  TBCS  project  had  already 
moved  forward  to  conduct  a  broad  mission,  function  and  task  analysis.  Therefore,  instead  of  addressing 
this  objective,  we  have  taken  the  opportunity  to  raise  a  number  of  issues  relevant  to  the  actual  ongoing 
HFE  task  analyses  and  data  modelling  being  conducted  for  TBCS,  which  are  important  and  are  not 
addressed  directly  in  the  reports.  These  issues  include:  the  nature  and  utility  of  the  HF  data  to  be 
generated  by  the  analyses,  the  need  to  consider  special  analyses  for  situation  awareness  and 
communication  issues,  since  these  are  core  processes  in  a  CCIS,  integration  with  other  CCIS  systems 
(scope  and  boundary  issues),  doctrine  and  team  considerations. 

Our  overall  conclusion  is  that  the  material  contained  within  the  reports  would  be  highly  beneficial  in 
providing  a  valuable  reference  source  to  the  military  community,  to  better  ensure  the  appropriate 
application  of  HFE  in  the  system  acquisition,  design  and  development  processes.  Widespread 
dissemination  of  the  material  covered  in  the  reports,  and  its  adoption  by  the  military  community,  are 
likely  to  reap  benefits  in  terms  of  ensuring  that  future  systems  meet  operational  needs,  and  are  built  in  a 
manner  than  mitigates  risks  to  operational  utility,  usability  and  overall  system  development  and  fielding 
costs. 
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1. Scope  and  Background 


This  work  was  commissioned  by  the  Defence  and  Civil  Institute  of  Environmental  Medicine 
DCIEM  as  part  of  a  standing  offer  contract  to  support  ongoing  Defence  Research  and  Development 
Branch  efforts  to  improve  the  application  of  human  factors  engineering  in  DND  CCIS  projects. 
Contract  No.  W771 1-6-7286/01-SRV.  The  Technical  Authority  is  Major  L.  Bossi  (DCIEM). 

1.1.  Objective 

The  objective  of  this  work  is  to  comment  upon  the  utility  of  two  related  reports1, 

1 .  Guidelines  for  Human  Engineering  Requirements  for  Canadian  Forces  Command  and 
Control  Information  Systems',  and 

2.  User  Manual:  Guidelines  for  Human  Factors  Engineering  Requirements  for  Canadian 
Forces  Command  and  Control  Information  Systems. 

The  analysis  of  the  reports  is  based  upon  the  following  perspectives 

•  The  approach  that  may  have  been  taken  to  TBCS  development  if  the  guidelines 
provided  in  the  preceding  reports  had  been  applied  from  the  inception  of  the  project. 

•  The  generic  usefulness  of  the  reports  to  the  acquisition  process  for  future  command 
and  control  information  systems  (CCIS) 

•  The  usefulness  of  the  reports  in  framing  the  Humansystems  Incorporated  (HSI)  report 
“Tactical  Battlefield  Command  System:  Human  Factors  tasks  and  risks  during  the 
procurement  cycle” 

An  initial  objective  was  to  recommend  the  immediate  elements  of  a  HFPP  that  would  be  required 
to  take  the  TBCS  project  forward  from  the  present  point.  However,  as  events  have  transpired,  the 
whole  scope  of  the  former  TBCS  project  has  been  re-evaluated,  and  project  management  has 
already  embarked  upon  a  course  of  implementing  some  of  the  core  components  of  a  HEPP,  as 
recommended  in  the  Engel  and  Townsend  reports. 


1  Note:  The  scientific  authority  indicated  that  a  third  related  document,  “Requirements  for  a  Human  Factors 
Engineering  Plan  for  Command  and  Control  System  Development',  was  not  be  reviewed  in  detail,  but  read 
as  relevant  contextual  material. 
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2.  Method 


The  various  sections  of  the  Engel  and  Townsend  reports  were  compared  with  MIL-STD-46855  and 
other  relevant  documents  to  identify  value  added  components  specific  to  CCIS  HFE  requirements. 
Relevant  experience  from  Human  systems  Incorporated  (HSI)  extensive  work  in  the  area  of 
development  and  evaluation  of  military  information  systems  was  also  used  as  a  context  for 
evaluating  detailed  aspects  of  the  report. 

The  utility  of  the  draft  Engel  and  Townsend  reports  for  developing  the  HSI  Report  “Tactical 
Battlefield  Command  System:  Human  Factors  tasks  and  risks  during  the  procurement  cycle”  was 
evaluated. 

The  possible  utility  of  the  Engel  and  Townsend  reports  (had  they  been  available  from  project  onset) 
to  the  TBCS  project  development  was  evaluated  through  interviews  with  the  TBCS  Project 
Director  (PD)  and  the  Leader,  Functions  Group,  Information  Systems  Technology  Group,  Defence 
Research  Establishment  Valcartier  (DREV). 

The  generic  usefulness  of  the  reports  was  also  explored  through  two  further  interviews  with  the 
Project  Manager  ARDS/ADM  and  the  HFE  Program  Manager/Govemment  Focal  Point  Displays 
and  Internal  Communication  System  R/SAOC  Modernisation  Program 
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3. Results 


The  results  are  organised  as  follows: 

•  Commentary  on  the  usefulness  of  the  “Guidelines”  report  for  developing  the  HSI 
outline  HEPP  for  TBCS  (section  3.1) 

•  Detailed  commentary  on  and  analysis  of  the  “Guidelines”  (section  3.2) 

•  Commentary  on  the  “User  Manual”  (section  3 .3) 

•  Specific  detailed  comments  on  the  “User  Manual”  (section  3.4) 

•  Interviews  with  the  project  managers  (section  3.5) 

The  initial  draft  of  the  report  was  received  by  HSI  in  mid  April,  and  this  draft  contained  only  part 
one  of  the  report  (Guidelines).  The  full  report  with  appendices  was  not  received  until  early  June. 

As  a  consequence,  and  given  the  end  of  April  deadline  for  delivery  of  the  draft  TBCS  HEPP  to 
DCIEM,  the  report  could  not  be  used  as  a  major  input  to  the  HSI  HEPP  preparation. 

3.1.  Usefulness  of  the  “Guidelines”  report 

The  “Guidelines”  report  provides  a  detailed  listing  of  the  major  components  that  would  be  expected 
to  form  the  core  of  an  HEPP  for  CCIS.  These  are  outlined  using  standard  HEPP  headings  and 
contain  a  clear  description  of  the  overall  processes  and  tasks  to  be  conducted.  An  appendix  provides 
detailed  data  item  descriptions  (DIDs)  for  each  of  the  core  elements  of  the  HEPP. 

These  aspects  of  the  Guidelines  assisted  HSI  by  validating  HSI’s  understanding  of  the  major  topic 
headings  for  the  HEPP.  This  information  was  organised  in  the  “Guidelines”  in  a  highly  coherent 
manner,  and  provided  important  cross  validation  for  HSI  with  others  sources  of  information 
concerning  HEPP  content  areas.  HSI  was  also  able  to  make  use  of  some  of  the  detailed  description 
of  several  of  the  human  factors  tasks  that  make  up  the  HEPP.  While  much  of  this  same 
information  was  obtainable  from  MIL-STD-46855,  the  organisation  of  the  “Guidelines”  resulted  in 
some  time  saving  in  assembling  the  information  into  a  coherent  process  model. 

Since  the  Guidelines  report  dealt  only  with  the  standard  HEPP  content  areas  this  limited  its  utility 
in  assisting  the  preparation  of  the  HSI  report,  which  dealt  with  some  broader  issues  relating  to  an 
overall  Project  Human  Factors  Plan  (PHFP).  This  meant  that  the  adaptation  of  the  Guidelines 
report  content  material  only  accounted  for  less  than  20%  of  the  required  effort  for  producing  the 
Humansystems  report. 

The  overall  PHFP  would  normally  include  issues  such  as  preparation  of  a  Statement  of 
Requirements  (SOR)  and  Statement  of  Work  (SOW),  Independent  Verification  and 
Validation  (IV&V),  training  and  job  re-design,  and  post-installation  activities  such  as 
review  of  issues  relating  to  doctrine  and  standard  operating  procedures.  This  PHFP  forms 
the  master  plan  for  co-ordinating  and  integrating  all  HF  activities  during  the  entire 
procurement  cycle.  It  also  spells  out  the  assignment  of  general  areas  of  HF  responsibility 
to  the  various  players,  which  include:  the  contractor,  the  PMO's  own  HF  advisor(s)  (in- 
house,  sub-contracted,  or  within  a  Defence  Research  Establishment  and  the  IV&V  team. 

Hence,  the  PHFP  is  at  a  higher  level  than  the  subsequent,  more  detailed,  HEPP  prepared  by 
the  contractor  to  cover  implementation  of  the  HF  responsibilities  assigned  by  the  PHFP. 
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These  issues  are  covered  by  Engel  and  Townsend  in  the  “Requirements”  Report  mentioned 
earlier  in  footnote  2.  Hence,  to  gain  the  necessary  appreciation  of  these  and  related  issues, 
the  reader  is  advised  to  consult  the  Requirements  report  as  a  companion  document  to  the 
User  Manual  and  Guidelines  Report. 

3.2.  Detailed  commentary  on  the  “Guidelines” 

For  the  most  part,  the  nature  of  this  commentary  is  in  the  form  of  a  critique  that  identifies  areas 
where  the  Guidelines  report  could  be  strengthened.  It  must  be  acknowledged  that  overall  this  report 
provides  sound  guidelines  and  principles,  with  which  HSI  agrees  and  supports.  For  practical  reasons 
we  do  not  enumerate,  or  comment  upon,  all  of  the  many  instances  of  the  material  with  which  we  are 
in  agreement.  Hence,,  the  result  of  this  approach  is  that,  on  balance;  the  review  may  appear  to  be 
more  generally  critical  rather  than  laudatory.  This  would  be  an  inappropriate  conclusion  to  draw, 
since  the  intention  of  this  review  is  to  improve  upon,  and  make  more  usable,  a  report  which  already 
goes  much  further  towards  meeting  the  needs  of  the  military  in  developing  a  CCIS  than  any  other 
documentation  that  is  currently  available,  and  of  which  HSI  is  aware. 

The  major  conclusions  from  the  analysis  of  the  Guidelines  report  are  as  follows. 

•  Overall  this  report  contains  much  of  the  information  contained  in  MIL-STD-46855. 

This  information  is  sensibly  re-organised  and  integrated  to  provide  better  information 
flow  than  the  original.  Generic  points  in  the  original  document  are  replaced  by  specific 
references  to  CCIS,  where  appropriate.  The  information  provides  a  potential  user  with 
a  succinct  but  detailed  summary  of  the  activities  outlined  in  MIL-STD-46855  that  are 
required  for  the  implementation  of  an  HEPP  into  the  project  acquisition  cycle. 

•  The  report  does  not  cross-reference  other  related  documents  developed  by  DCIEM 
which  are  of  relevance  to  the  development  of  the  HF  plan  (e.g.  references  5,  7).  These 
provide  additional  information  that  a  PM  would  need  to  understand  in  applying  the  HF 
plan. 

•  While  the  report  is  well  organised,  it  may  not  provide  a  structure  that  maps  the  HEPP 
in  an  obvious  way  to  the  understanding  which  military  users  may  bring  concerning  the' 
overall  systems  engineering  process.  This  may  be  exacerbated  because  some  potential 
users  may  have  little  knowledge  of  human  factors  beyond  Staff  College,  while  others 
may  have  seen  human  factors  only  applied  late  in  system  development.  Few  will  have 
an  immediate  appreciation  of  the  full  scope  of  an  integrated  HFE  effort  and  will  need 
some  supporting  arguments  to  “buy  in”  to  the  proposed  approach.  In  this  respect,  the 
provision  of  some  elements  of  the  separate  Engel  and  Townsend  “Requirements” 
report  which  argues  the  case  for  integrating  HFE  into  systems  development  would  be 
useful  in  providing  this  additional  background  familiarisation. 

•  A  section  on  measures  of  performance  and  effectiveness  is  provided  but  this  does  not 
incorporate  specific  measures  of  C2  effectiveness  that  would  be  appropriate  for  CCIS. 

•  The  suggestion  is  made  in  this  section  of  the  report  that  subjective  rating  scales  should 
not  be  used  as  proxies  for  objective  empirical  measures  of  performance.  However,  this 
is  a  debatable  issue  given  the  centrality  of  subjective  measures  to  the  assessment  of 
workload,  which  is  a  critical  performance  indicator  for  C2  processes.  Also,  subjective 
estimates  form  the  basis  for  some  of  the  simulation  parameters  that  underlie  modelling 
of  tasks  and  network  simulations  of  function  allocation. 
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•  A  useful  section  on  decision  support  requirements  is  provided,  but  this  is  not  tailored 
specifically  to  CCIS  needs. 

•  To  complement  the  sections  on  design  for  decision  support,  which  is  a  clear  core 
element  for  a  successful  CCIS,  it  would  have  been  appropriate  to  incorporate  issues 
relating  to  design  support  for  situation  awareness 2  and  communication  effectiveness. 

•  Some  improvements  could  be  made  to  section  (6.4.3)  which  deals  with  tailoring  to 
meet  the  scope  and  needs  of  the  acquisition  project.  This  section  breaks  down  the 
application  of  HF  activities  into  phases  of  analysis,  human  factors  in  design  and  human 
factors  in  test  and  evaluation.  The  usefulness  of  this  section  could  be  augmented  by 
providing  an  overarching  framework  to  guide  the  reader  on  how  the  sequential  nature 
of  each  of  the  critical  human  factors  activities  map  onto  the  overall  system  acquisition 
plan.  This  section  should  be  cross-referenced  to  the  general  sequential  outline  of  the 
global  HFE  plan  (GHFEP)  that  is  found  in  the  “Requirements”  report. 

Overall,  the  Guidelines  report  provides  an  excellent  and  well  organised  outline  of  the  typical 
activities  and  associated  documentation  that  would  be  expected  to  form  the  basis  of  an  effective 
HEPP  designed  for  use  in  developing  and  acquiring  military  information  systems. 

In  summary,  the  areas  where  the  utility  of  the  report  could  be  enhanced  are  as  follows,  the  report 
should 

•  Provide  a  clearer  outline  of  the  sequential  steps  in  implementing  a  HEPP  as  they  relate 
to  the  ongoing  project  acquisition  cycle. 

•  Provide  information  on  those  human  factors  tasks  that  must  be  considered  prior  to  the 
point  at  which  an  HEPP  usually  comes  into  force. 

•  Address  specific  considerations  that  should  be  applied  to  an  HEPP  that  would  be 
developed  for  a  CCIS  application.  For  example,  make  references  to  CCIS  specific 
measures  of  performance,  unique  operating  environments,  C2  organisational  structures, 
issues  of  doctrine  or  standard  operating  procedures,  and  systems  integration  both 
horizontally  and  vertically  within  military  command  levels. 

•  Add  sections  on  design  for  situation  awareness  and  communication. 

•  Provide  a  risk/benefit  analysis,  of  performing  each  of  the  HEPP  major  tasks  in  a  CCIS 
environment3. 

•  Provide  better  links  to  the  mental  model  that  PM’s  with  varying  experience  will  bring 
to  the  report. 

•  Provide  improved  signposts  for  PM’s  who,  because  of  circumstances,  cannot  integrate 
the  HF  plan  from  the  project  outset. 

It  should  be  recognised  that  some  of  the  suggestions  made  above  may  go  beyond  the  specific 
mandate  DCIEM  provided  for  the  preparation  of  the  reports  by  Engel  and  Townsend,  which 
required  that  the  contractor  was  to  observe  and  review  the  application  of  HFE  in  the  ARDS/ADM 


The  term  situation  awareness  is  used  frequently  in  this  report.  Since  military  and  scientific  personnel 
reading  this  report  may  interpret  this  concept  in  different  ways,  a  brief  overview  of  the  concept  of  situation 
awareness  is  provided  in  Annex  B. 

3  This  suggestion  arises  both  from  an  expressed  guideline  provided  by  TBCS  project  management  to  HSI  in 
preparation  of  the  draft  HEPP,  as  well  as  from  points  subsequently  arising  from  interviews. 
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project  and  "produce  a  final  report  and  a  generic  HFE  plan  for  CCIS  reflecting  the  ARDS  ADM 
experience" 

3.3.  Commentary  on  the  “User  Manual” 

Several  major  issues  are  outlined  below. 

3.3.1.  Consideration  of  additional  forms  of  HFE  analysis 

As  in  most  generic  HEPPs,  the  User  Manual  makes  a  strong  case  for  workload  analysis  as  a 
primary  means  of  identifying  both  the  appropriateness  of  the  design  and  the  optimal  allocation  of 
functions.  However,  workload  analysis  is  a  tool  that  provides  little  initial,  specific  feedback  for 
system  developers  in  terms  of  directly  identifying  the  underlying  causes  of  inappropriate  workload. 
For  example,  when  using  modelling  methods,  a  typical  practice  is  to  grossly  manipulate  allocations 
of  functions,  or  modify  processes  or  their  task  sequences,  to  determine  their  effects  on  workload. 
However,  such  an  approach,  which  may  be  both  costly  and  time  consuming,  may  in  the  end  fail  to 
identify  the  exact  factors  in  the  system  that  drive  inappropriate  workload.  In  particular,  it  is  not 
clear  how  the  overall  analysis  of  workload,  or  its  typical  breakdown  into  perceptual,  cognitive  and 
psychomotor  components,  provides  insight  into  the  detailed  issues  of  operator-system  performance 
that  need  to  be  understood  from  a  design  perspective. 

We  believe  that  workload  is  a  product  of  several  intervening  drivers  of  performance,  which  include 
factors  such  as  situation  awareness,  decision  making  and  communication  effectiveness.  Hence 
from  a  design  analysis  perspective,  it  makes  sense  to  analyse  how  the  potential  design  may  impact 
upon  each  of  these  factors.  The  importance  of  these  areas  to  overall  C2IS  effectiveness  were 
substantiated  in  an  earlier  HSI  report  commissioned  by  DCIEM,  (reference  5). 

Hence,  it  is  recommended  that  the  analysis  of  situation  awareness,  decision  making  and 
communication  be  specified  as  core  required  components  of  an  HEPP  for  all  CCIS  to  complement 
workload  analysis. 

A  further  consideration  in  the  use  of  workload  and  other  HF  analyses,  which  are  based  closely  on 
existing  systems  and  procedures,  is  that  the  analyses  may  have  limited  generality,  and  hence 
usefulness,  in  informing  requirements  specification  or  detailed  design  for  “revolutionary”  systems. 
That  is,  newly  emerging  systems  may  take  advantage  of  new  technologies,  may  need  to  reflect 
changes  in  doctrine  and  may  need  to  accommodate  potential  reductions  in  personnel  resources. 

Thus,  drivers  of  workload  and  existing  system  function  allocations  may  have  no  direct  counterpart 
in  next  generation  systems.  In  such  cases,  it  would  seem  advisable  to  recommend  that  HF  analyses 
concentrate  on  the  higher  order  mission  functions  and  examine  the  generic  information, 
communication  and  decision  needs  for  potential  users  of  such  systems. 

3.3.2.  Risks,  benefits  and  local  appropriateness  of  the  various  HEPP 
activities. 

While  the  report  contains  a  complete  inventory  of  all  of  the  elements  of  an  HEPP,  there  are  no 
guidelines  provided  for  the  inexperienced  user,  of  the  various  trade-offs  that  may  need  to  be  made  in 
applying  an  HEPP  under  typical  system  development  constraints.  The  report  contains  a  useful  and 
comprehensive  section  on  tailoring  the  plan  for  specific  systems  engineering  and  project  management 
methods,  but  the  report  does  not  address  difficult  issues  concerning  the  ultimate  value  of  each  of  the 
human  factors  analyses.  Hence,  it  gives  the  impression  that  all  aspects  of  the  plan  need  to  be 
implemented  no  matter  what  the  local  context  (notwithstanding  the  section  of  the  report  that  deals 
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with  small  applications  development).  Sections  of  reference  7  could  be  cross-referenced  or  integrated 
into  a  final  document,  to  provide  the  PM  with  a  greater  understanding  of  risks  and  benefits. 

More  generally,  summary  statements  should  be  provided  of  the  major  benefits  to  be  gained  from 
implement  each  component  of  the  HEPP  and  the  corollory  of  the  types  of  risk  that  will  be  incurred 
if  the  activity  is  not  carried  out. 

3.3.3.  Specificity  in  the  measures  of  system  effectiveness 

This  section  would  be  more  useful  if  an  inventory  of  measures  of  effectiveness  were  provided  for 
users  of  the  manual  who  are  unfamiliar  with  the  appropriate  measures,  nor  have  the  ability  or 
background  to  develop  them  by  themselves.  In  a  CCIS  environment  there  are  a  limited  number  of 
practical  measures  that  provide  key  indicators  of  system  performance.  These  have  been  documented 
by  several  sources  and  have  been  integrated  and  summarised  in  references  5  and  6. 

3.3.4.  Specific  guidance  on  how  to  use  Human  Engineering  data  to 
influence  statements  of  requirements  and  specific  details  of  design. 

Following  the  various  HF  analyses  that  are  conducted  as  part  of  the  HEPP,  a  major  task  for  the  PM 
will  be  to  use  the  resulting  HF  data  to  influence  system  specification  and  in  some  cases  detailed 
design  requirements.  This  can  be  a  daunting  task,  since  it  will  not  be  intuitively  obvious  to  PM’s 
on  how  to  do  this.  HF  task  analyses  and  modelling  will  typically  produce  volumes  of  task 
inventories,  operational  sequence  diagrams,  and  hundreds  of  graphs  of  modelled  data.  Somehow 
these  data  must  be  translated  into  information,  which  in  turn  must  be  applied  to  requirements 
definition.  Since  this  process  is  not  taught  in  staff  college,  and  most  PM’s  will  have  little  or  no 
experience  in  how  to  embark  upon  such  a  task,  some  detailed  guidance  should  be  provided  on  how 
this  process  is  to  occur.  Indeed,  it  might  be  useful  to  provide  an  annex  to  the  report  that  provides 
sample  data  from  a  limited  military  context,  to  illustrate  the  nature  and  form  of  the  HF  data  that  are 
actually  generated  by  mission,  function  and  task  analysis.  The  danger  exists  that  if  the  PM  cannot 
understand  the  data  generated,  or  comprehend  its  relevance  to  system  design,  or  translate  the  data 
into  a  requirements  statement,  then  the  entire  HF  analytical  process,  which  produce  such  outputs, 
may  lose  credibility  in  its  claim  for  a  central  role  in  the  general  systems  engineering  analysis  of  a 
new  system. 

3.3.5.  Consideration  of  the  military  “team”  environment 

One  of  the  major  differences  between  a  typical  civilian  and  a  military  information  system,  is  the 
added  dimension  of  the  integral  nature  of  team  functions  to  achieving  military  C2  objectives.  The 
traditional  HEPP  does  not  take  into  account  the  additional  user  requirements  that  may  be  arise  when  a 
tightly  integrated  team  must  function  with  a  shared  information  system.  Emerging  areas  such  as  team 
situation  awareness  and  implicit  communication  modes  will  require  some  context  specific  research  to 
be  performed  following  the  initial  function  analysis.  In  this  way,  issues  may  be  identified  relating  to 
the  specification  of  requirements  that  are  unique  for  team  operations.  Considerations  of  team 
operations  will  also  have  a  major  impact  on  test  and  evaluation  and  design  feedback. 

3.3.6.  Language  confusions 

In  the  experience  of  HSI,  there  are  frequent  miscommunications  between  the  various  parties 
engaged  in  the  system  development  process  concerning  core  concepts,  because  of  local  differences 
in  interpretation  of  meanings.  One  frequent  area  of  misunderstanding  occurs  over  the  issue  of 
“criteria”  and  related  terms;  few  people  understand  the  clear  distinction  between  criteria,  measures, 
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and  standards.  Hence,  it  would  be  useful  to  have  a  glossary  of  technical  terms  with  definitions 
provided  for  naive  (or  non-technical)  readers.  Other  confusions  may  occur  between  HF  personnel, 
military  SMEs  and  software  developers  in  the  usage  of  terms  such  as  function  and  tasks,  and  the 
concept  of  situation  awareness. 

3.4.  Specific  detailed  comments  on  the  “User  Manual” 

The  numbers  assigned  to  the  comments  below  refer  to  paragraph  numbers  in  the  user  manual. 

2.2  Some  useful  references  are  missing,  such  as  the  ANSI-HFS  guidelines  for  workstation 

design  and  the  HSI  report  on  framework  for  evaluation  and  measures  of  performance  and 
effectiveness  for  C2  systems. 

2.0  An  argument  may  be  made  against  the  statement  that  the  general  aim  of  HFE  is  to  make 

systems  easy  for  people  to  use.  There  are  a  number  of  other  design  goals  such  as  improved 
system  effectiveness,  efficiency,  and  safety. 

3.3.1  Table  1 .  The  table  shows  that  the  primary  assessment  of  design  effectiveness  is  the 
measurement  of  speed  and  accuracy  in  doing  high  level  cognitive  tasks.  However,  from  a 
design  feedback  or  improvement  perspective,  such  measures  of  overall  task  completion 
time,  for  example,  may  carry  insufficient  diagnosticity  of  design  problems.  Measures  of 
“timeliness”  and  “completeness”  may  be  more  appropriate  in  some  circumstances. 
Therefore,  it  will  be  critical  to  ensure  that  the  performance  measures  are  directed  at  an 
appropriate  level  of  behaviour  to  reflect  the  specific  information  processing  that  the  system 
must  support.  It  may  also  be  necessary  to  decompose  some  high  level  cognitive  tasks  into 
some  of  their  constituent  elements.  An  example  of  this  would  be  to  use  separate 
assessments  of  the  ability  of  a  design  to  support  all  three  levels  of  situation  awareness; 
namely  (i)  rapid  detection  of  new  information,  (ii)  integration  and  comprehension  of 
information  into  a  coherent,  internal  representation  and  (iii)  projection  of  the  representation 
into  the  future,  to  anticipate  further  information  needs  and  decisions  (see  Annex  B.  for  an 
overview  of  the  concept  of  situation  awareness). 

3.3.2  Four  of  the  major  differences  between  the  application  environments  of  a  CCIS  and  MIS, 
that  should  be  emphasised  here  are: 

•  the  need  for  the  user  of  a  CCIS  to  maintain  high  levels  of  situation  awareness 

•  the  centrality  of  an  effective  communication  system  to  achieve  mission  goals 

•  the  generally  high  level  of  background  message  traffic  that  competes  for 
attention 

•  the  general  tendency  for  users  of  the  CCIS  to  be  operating  in  a  multi-tasking 
environment. 

3.0  Not  much  detail  is  provided  on  the  forms  of  HF  analysis.  Mission,  task  and  function 

analyses  tend  to  be  descriptive  rather  than  analytical  in  terms  of  their  output.  Workload 
estimates  are  the  only  true  analytical  method  proposed  here.  Hence,  there  is  a  need  to 
show  the  requirement  for  supplemental  analyses  of  situation  awareness,  decision  making 
and  communication  requirements. 

4. 1 .3  The  implementation  of  other  complementaiy  procedures  to  basic  task  analysis  should  be 
considered.  These  may  provide  additional  sensitivity  and  validity,  and  could  provide  the 
level  of  diagnostic  accuracy  that  will  be  required  for  design  specification.  Some 
consideration  should  be  given  to  expanding  the  traditional  view  of  task  analysis,  as 
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outlined  in  the  report,  to  incorporating  some  of  the  ideas  outlined  in  NATO  STANAG 
3994.  (In  particular,  the  references  to  analysis  for  situation  awareness,  information 
requirements  and  communication). 

4.2.1  The  military  reader  will  not  find  a  strong  case  here  for  the  value-added  component  for 
HFE,  since  the  identification  of  requirements  the  HF  part  of  the  analysis  is  not  made 
explicit. 

Perhaps  a  better  approach  would  be  to  suggest  an  integrated  team  with  army  SMEs, 
developers  and  HF  specialists.  More  stress  should  be  given  in  the  report  to  the  particular 
importance  of  HF  specialists  in  providing  HFE  expertise  in  analytical  methods,  their 
unique  knowledge  of  available  and  relevant  the  human  performance  data,  and  their  crucial 
role  in  helping  translate  data  from  HF  analyses  into  requirements  and  design  guidance. 

It  is  not  made  clear  how  user  performance  requirements  can  be  determined  from  the 
methodology  described  here.  In  particular,  how  would  this  approach  assist  in  determining 
performance  requirements  for  next  generation  systems?  The  role  of  experienced  HFE 
personnel  in  assisting  the  translation  from  HFE  analyses  to  requirements  should  be  noted 
here.  This  may  be  particularly  important  for  systems  that  represent  a  “revolutionary” 
design  potential  as  opposed  to  an  “evolutionary  “  approach. 

One  approach  would  be  to  use  the  data  obtained  from  MOEs  to  assist  in  setting 
performance  requirements,  however  the  report  does  not  provide  any  detailed  MOEs  for 
critical  areas  of  which  influence  the  performance  of  a  CCIS. 

The  time  required  for  developing  capabilities  and  deficiency  descriptions  is  probably  too 
short  and  unrealistic,  based  upon  HSI’s  experience  across  a  number  of  project  domains. 

4.2.2  The  section  on  function  allocation  only  deals  with  the  traditional  area  of  human-machine 
allocation  and  ignores  the  question  of  function  allocation  across  members  of  the  typical 
military  team  in  conjunction  with  the  system. 

4.2.6  Consistent  with  the  above  general  major  comments,  it  is  recommended  that  the  list  of  HFE 
questions  in  the  design  review  process  for  a  CCIS  be  expanded  as  follows. 

User  situation  awareness:  will  the  interface  and  system  design  allow  the  user  to  rapidly 
detect  changes  in  information,  integrate  such  information  into  the  current  (battlefield) 
picture  and  project  how  the  picture  may  change  in  the  foreseeable  future?  Will  the 
interface  and  system  design  allow  the  user  to  integrate  map  and  communication  based  data 
from  local  and  more  global  contexts  into  the  relevant  picture  of  the  battlefield? 

User  decision  making-,  does  the  system  design  provide  the  user  with  the  necessary 
information  at  the  appropriate  time  and  formatted  in  a  meaningful  manner  to  facilitate  both 
slow  paced  planning,  or  faster  paced  tactical,  decision  making? 

User  communication :  does  the  system  design  allow  users  to  communicate  effectively  and 
accurately  with  a  minimum  of  effort?  Does  the  system  provide  a  means  for  a  meaningful 
integration  of  information  from  different  communication  sources  and  modes?  Does  the 
system  allow  the  user  to  manage  communication  effectively  with  a  high  volume  of 
message  traffic? 

4.2.6. 5  Specific  measures  of  effectiveness  should  be  provided  here.  This  is  certainly  feasible  since 
there  is  a  core  set  of  generic  functions  and  processes  that  determine  the  overall 
effectiveness  of  most  CCIS  (reference  #5). 
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4.3 .3.1  Preceding  comment  also  applies  to  the  specification  of  test  measures. 

4.3.3 .3  Examples  of  valid/invalid  and  reliable/unreliable  measures  could  be  provided  here  for  a 
person  unfamiliar  with  these  concepts. 

4.3.4  Note  in  Table  2  that  an  advanced  form  of  storyboard  could  be  a  screen-based  example  of  a 
piece  of  system  functionality,  produced  through  rapid  prototyping. 

4.3.6  Some  additional  advice  should  be  provided  on  how  to  achieve  levels  of  reliability  in  testing 
when  the  typical  constraints  found  in  a  military  environment  (e.g.  availability  of  test 
subjects)  compromise  the  requirements  for  statistical  significance. 

4.4. 1  One  issue  that  should  be  mentioned  here,  and  which  may  be  typically  relevant  for  most 
new  installations  of  a  CCIS,  concerns  the  integration  of  the  new  system  with  existing, 
standalone  systems,  with  which  users  must  continue  to  interact  to  achieve  mission  goals. 
Specific  examples  could  be  provided  in  the  task  analysis  and  function  analysis  sections 
(4.1.2,  4.1.3)  to  show  how  these  analyses  take  into  account  the  future  interaction  between 
the  proposed  CCIS  and  existing  systems. 

3.5.  Interviews  with  the  TBCS  project  personnel. 

Interviews  were  conducted  with  four  individuals,  each  of  which  had  a  unique  and  different 
perspective  based  upon  experiences  in  the  management  of  new  CCIS.  The  individuals  were: 

1.  Leader,  Functions  Group,  Information  Systems  Technology  Group  DREV 

2.  TBCS  Project  Director.  Written  comments  were  also  received  from  the  TBCS  Staff 
Officer. 

3.  Project  Manager  ARDS/ADM. 

4.  HFE  Program  Manager/Govemment  Focal  Point  Displays  and  Program  Manager  for 
HFE  for  the  Internal  Communication  System  R/SAOC  Modernisation  Program. 

The  format  of  the  interview  was  semi-structured  with  the  major  questions  provided  in  advance  to 
the  interviewees.  The  major  issues  raised  in  each  of  the  interviews  are  outlined  below  and  grouped 
into  themes.  To  reduce  any  overhead  in  translating  the  transcripts  into  a  more  flowing  narrative, 
the  comments  are  provided  in  “note”  format.  A  more  detailed  transcription  of  each  entire  interview 
is  presented  in  Annex  A. 

The  major  themes  chosen  for  analysis  were: 

•  Report  style  and  organisation 

•  HFE  Process  and  method 

•  How  the  information  contained  in  the  reports  might  have  influenced  the 
TBCS/Chameleon  approach  (or  relevant  system  for  the  interviewee  in  question) 

•  How  the  reports  provide  insight  into  the  major  steps  for  the  next  phases  of  the  project 

•  Comments  on  specific  aspects  of  the  reports  where  improvements  can  be  made. 
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3.5.1.  Interview  with  the  Leader,  Functions  Group,  Information  Systems 
Technology  Group  DREV. 

Report  style  and  organisation 

There  is  value  in  having  the  reports  organised  into  a  Guidelines  Document  and  User  Manual. 

Many  military  users  will  probably  just  opt  for  the  brevity  of  the  Guidelines.  However,  for  a 
military  user,  unfamiliar  with,  and  possibly  less  convinced  of  the  need  for,  an  HFE  plan,  there  is  a 
need  to  provide  an  introductory  section  in  the  Guidelines  report.  This  section  should  outline  the 
background  rationale  for  human  factors  being  a  critical  aspect  of  the  general  systems  engineering 
program  and  effort.  This  introduction  and  rationale  would  also  be  needed  in  the  case  of  a 
contractor  who  was  unfamiliar  with  the  full  scope  of  an  HFE  effort. 

The  level  of  technical  writing  is  appropriate  for  military  PM’s  and  defence  scientists. 

The  reports  are  organised  in  a  meaningful  and  coherent  manner. 

The  surface  style  of  the  reports  is  too  “dense”  and  does  not  show  good  design  principles.  In 
particular  there  are  long  sections  of  unbroken  text  which  may  be  difficult  to  digest  for  the  average 
PM  who  is  hard  pressed  for  time.  The  greater  use  of  text  and  graphic  formatting  would  help  to 
alleviate  the  impression  of  density  and  make  it  easier  for  the  reader  to  stay  with  complex  issues. 

A  major  concern  was  expressed  over  the  small  number  of  specific  military  examples.  Where  they 
were  used,  for  example  in  the  section  on  deficiencies  in  requirements  statements,  they  made  the 
concepts  much  more  comprehensible.  The  use  of  a  specific  example  throughout  the  document  is 
recommended  to  assist  the  reader  in  understanding  more  difficult  concepts  such  as  function 
allocation  and  operational  sequence  analysis.  Thus,  a  concept  such  as  the  generation  of  an  Op 
Order  could  be  viewed  from  the  perspective  of  mission  analysis,  task  and  function  analysis  etc. 
Further  it  was  suggested  that  more  examples  should  be  provided  to  illustrate  DIDS. 

HFE  Process  and  method 

A  major  concern  was  expressed  over  the  fact  that  the  User  Manual  did  not  clearly  show  in  detail 
how  the  HFE  program  was  integrated  with  other  systems  engineering  activities.  The  impression 
was  given  that  the  User  Manual  seems  to  be  strongly  pushing  the  insertion  of  a  new  “block”  into 
normal  systems  engineering  process  approach.  This  may  be  perceived  as  threatening  to  systems 
engineers  and  PM’s.  The  User  Manual  is  seen  as  pushing  the  “independent  HFE  model”  too  hard. 
The  opinion  was  given  that  HFE  is  best  integrated  during  the  development  of  functional 
requirements.  Whereas  the  User  Manual  appears  to  imply  that  HFE  analysis  is  done  beforehand, 
instead  of  in  parallel  with  other  technical  analysis.  Further,  the  User  Manual  does  not  make  it  clear 
how  HFE  is  integrated  into  life  cycle  systems  development.  (HSI  comment:  however,  the  bottom 
paragraph  of  page  39  clearly  shows  how  the  HFE  control  loop  can  be  successfully  integrated  into 
both  iterative  and  parallel  approaches  during  the  design  phase  of  system  development). 

There  was  strong  disagreement  with  the  general  direction  of  the  HFE  process  model  proposed  in 
many  areas.  The  User  Manual  should  provide  more  details  on  iterative  approaches  as  opposed  to 
what  is  perceived  as  the  waterfall  approach  that  is  being  promoted.  For  emerging  technologies  the 
proposed  model  is  inappropriate,  since  it  does  not  allow  for  some  preliminary  iterations  through 
actual  design  implementations.  There  was  disagreement  with  the  approach  that  you  can  first  do 
task  analysis,  then  validate  and  then  design.  As  a  consequence,  the  view  was  expressed  that  the 
User  Manual  should  show  more  clearly  how  HFE  activities  interact  with  ongoing  prototype 
development.  Further,  you  cannot  write  HFE  deficiencies  without  knowing  what  technologies  are 
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feasible.  Essentially,  a  hybrid  approach  was  advocated  that  combines  prototyping  with  more 
formal  HFE  methods-  quote  “analyse  a  bit,  design  and  develop  a  bit,  test  a  bit”. 

How  the  information  contained  in  the  reports  might  have  influenced  the 
TBCS/Chameleon  approach. 

The  initial  approach  taken  to  TBCS  development  was  based  upon  an  LFCS  process  model,  which 
uses  the  staff  planning  process  as  a  core  baseline  concept  (taught  at  Staff  College).  This  was 
developed  into  Chameleon  to  provide  a  demonstration  vehicle  for  user-oriented  concepts  and 
functions.  The  lack  of  task  analysis  was  discovered  later  in  the  process,  as  was  the  need  to  develop 
scenarios  based  upon  mission  analysis.  Later  the  team  came  to  the  view  that  task  analysis  needs  to 
be  a  part  of  iterative  prototyping. 

The  major  effect  the  reports  would  have  had  on  the  actual  approach  taken  was  in  the  area  of 
preliminary  mission  and  task  analysis.  This  could  have  mitigated  some  cost,  schedule  and 
operational  applicability  risks  in  terms  of  better  understanding  the  needs  of  the  combat  team 
members  in  the  context  of  representative  mission  scenarios.  Notwithstanding  this,  the  fundamental 
approach  may  not  have  changed  very  much,  since  there  is  still  a  need  to  demonstrate  a  prototype 
and  then  validate  this  through  a  scenario,  prior  to  more  substantive  HFE  analysis  activities,  such  as 
function  allocation  and  OSD’s.  However,  a  prior  mission  analysis  would  allow  priorities  for  the 
first  prototype  to  be  more  easily  identified. 

How  the  reports  provide  insight  into  the  major  steps  for  the  next  phases  of  the 
project 

The  most  important  process  to  follow  next  is  to  conduct  an  appropriate  mission  and  task  analysis. 
This  should  be  followed  with  a  review  and  analysis  of  function  allocation,  and  then  a  comparison 
should  be  made  of  the  product  of  this  analysis  with  the  functionality  of  Chameleon  prototypes  to 
date.  The  reports  underline  the  need  to  ensure  that  the  next  implementation  of  the  Chameleon 
prototype  development  incorporates  a  broader  spectrum  of  HFE  considerations,  in  particular  the 
need  for  formal  testing.  The  reports  suggest  that  a  next  obvious  step  would  be  to  generate  a  special 
set  of  HFE  requirements  as  part  of  a  functional  SOR.  There  will  also  be  a  need  to  integrate  HFE 
into  predefinition  activities,  for  storyboarding  concepts,  technology  demonstrations,  or  other 
simulation-based  acquisition  approaches. 

Comments  on  specific  aspects  of  the  reports  where  improvements  can  be  made. 

The  reports  do  not  spell  out  the  precise  make  up  of  HFE  team  and  individual  responsibilities. 

The  User  Manual  needs  to  show  how  the  HFE  activities  are  better  fused  throughout  the 
design/development  cycle.  It  implies  separate  but  connected  processes-but  a  more  integrated 
approach  should  be  spelled  out  in  full  detail. 

The  User  Manual  provides  some  idea  of  the  scope  of  the  HFE  effort,  but  not  in  enough  detail. 
Providing  an  approximate  estimate  of  the  cost  of  each  activity  would  not  be  particularly  useful. 

The  time  required  for  developing  requirements  from  analysis  is  likely  to  be  much  more  extensive 
than  is  reflected  in  the  User  Manual. 

The  risks  associated  with  not  doing  each  component  of  the  HFE  are  not  provided,  except  the  User 
Manual  does  make  clear  the  risks  associated  with  not  performing  better  initial  analysis  and 
requirements  definition  early  in  project  development.  It  would  be  useful  to  provide  more  tangible 
examples  of  HFE  work  and  to  show  how  risk  is  mitigated. 
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Insufficient  detail  is  provided  of  the  type  of  data  that  is  generated  by  some  of  the  HF  activities,  e.g. 
operational  sequence  analysis  and  function  allocation  and  modelling. 

No  guidelines  are  provided  on  how  the  output  of  HFE  analyses  can  be  specifically  used  for  the 
creation  of  requirements  statements,  particularly  with  respect  to  critical  system  performance  areas. 

4.2.2  Function  allocation:  the  question  was  raised  as  to  whether  this  systematic  approach  is 
practicable  in  army  CCIS  environment?  An  analysis  of  function  allocation  was  not  performed  in 
the  development  of  the  Chameleon  product,  but  the  reports  provided  a  good  perspective  on  how 
this  issue  might  have  been  approached  in  a  more  systematic  manner.  However,  analysis  should  go 
hand  in  hand  with  developing  the  technology  available,  and  is  complex  to  implement  (more 
information  should  be  provided  on  technology  constraints  and  enablers).  This  section  is  too  remote 
from  technology  development  and  needs  to  be  better  integrated. 

4.2.3  Was  surprised  to  learn  that  there  was  “no  fixed  procedure  for  translating  HFE  data  into 
design  requirements”.  Some  examples  of  types  of  approaches  should  be  provided  here. 

4.2.6.4  Would  like  to  see  more  detailed  description  of  the  actual  simulation  process  and  the  level 
of  effort  required.  Believes  that  while  this  may  be  useful  for  the  technological  component  it  is  not 
clear  how  to  model  and  integrate  the  user  component. 
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3.5.2.  Summary  of  Interview  with  the  TBCS  Project  Director. 

Summary  of  major  points  arising  from  interview  with  the  TBCS  PD  and  written  comments 
provided  by  the  TBCS  PMO  Staff  Officer. 

Background 

There  was  only  sufficient  time  available  to  study  the  Guidelines  Document  in  detail  and  to  quickly 
skim  the  User  Manual. 

The  TBCS  Project  Director  pointed  out  what  he  considered  to  be  an  important  distinction  between 
two  military  information  systems.  The  larger,  the  Command  and  Control  Information  System, 
comprises  the  entire  operating  environment  of  the  army  and  incorporates  information  needs  at  all 
levels,  as  well  as  issues  relating  to  training,  doctrine  and  operating  procedures.  The  primary 
purpose  of  the  CCIS  is  to  support  the  commander  (HSI  note:  it  is  not  clear  whether  this  refers  to  the 
commander  specifically,  or  the  command  team).  Below  the  CCIS  are  smaller  sub-systems,  which 
comprise  the  C2IS.  These  sub-systems  are  largely  concerned  with  hardware  and  software  to  meet 
the  information  needs  at,  for  example,  the  brigade,  combat  team  or  individual  soldier  levels.  At  the 
level  of  the  CCIS,  system  analysis  will  identify  several  types  of  deficiencies.  Some  will  require  a 
hardware  solution,  others  a  procedure  solution  or  changes  to  command  structure  and/or  doctrine. 

In  general,  the  reports  support  analysis  only  at  the  C2IS  level  and  do  not  deal  with  human  factors 
contributions  at  the  CCIS  level,  and  should  be  expanded  to  incorporate  this  wider  domain. 

Report  style  and  organisation 

Overall,  the  information  in  the  reports  is  densely  packed.  The  average  “hard  pressed”  PM  would 
be  unlikely  to  take  the  time  to  try  to  find  the  needed  piece  of  information  under  typical  time 
pressure.  The  format  should  be  revised  to  make  the  search  for  relevant  information  easy  to  achieve 
(hypertext  links  were  suggested).  The  further  point  was  made  that  in  this  day  of  widespread 
electronic  documentation,  military  personnel  would  rarely  set  aside  a  few  hours  to  read  through 
paper  documentation. 

The  two  reports  need  better  cross-links. 

The  reports  are  organised  in  a  manner  that  the  average  PM  would  readily  comprehend  and  the  level 
of  technical  language  was  appropriate.  However,  for  a  reader  with  no  background  in  HF,  both  the 
PD  and  Staff  Officer  believe  that  some  prior  background  training  would  be  necessary  (he  suggested 
a  CD-ROM  based  course).  The  Staff  Officer  noted  that  a  novice  PM  would  have  some  difficulty  in 
following  the  overall  process. 

HFE  Process  and  method 

The  process  and  method  were  outlined  very  clearly,  as  were  the  links  between  the  HEPP  and  the 
overall  systems  engineering  plan.  The  reports  outline  in  sufficient  detail  the  major  HF  tasks,  their 
scope  and  the  overall  responsibilities  for  conducting  the  component  elements  of  the  HEPP. 
However,  much  more  detail  should  be  provided  on  scope  and  costs  to  help  trade-off  issues.  It  was 
noted  that  some  of  this  information  was  available  through  the  tables  and  annexes  in  the  HSI  report 
(reference  #7). 

The  breakdown  by  phase  of  development  was  useful,  as  were  the  detailed  examples  of  the  DIDs. 
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How  the  information  contained  in  the  reports  might  have  influenced  the 
TBCS/Chameleon  approach 

The  information  in  the  reports  reinforces  and  supports  the  view  that  the  PD  brought  to  this  project. 
At  that  time,  the  existing  system  development  plan  was  already  to  far  along  to  make  radical 
changes.  In  general,  the  ongoing  approach  was  disjointed.  Had  the  PD  been  able  to  influence  the 
direction  of  the  project  at  the  outset,  he  would  have  relied  extensively  on  the  reports,  which 
contains  essential  and  valuable  information  on  how  the  human  factors  and  systems  planning  should 
properly  evolve,  and  how  the  HF  effort  should  form  the  early  core  of  system  development. 

Initial  tasks  in  the  system  development  would  have  concentrated  upon  mission  analysis  and 
scenario  development,  instead  of  developing  functionality. 

How  the  reports  provide  insight  into  the  major  steps  for  the  next  phases  of  the 
project. 

Essentially,  the  most  important  information  in  the  reports  is  the  direction  they  provide  concerning 
the  immediate  way  to  proceed  for  developing  requirements  statements  for  the  next  generation 
system.  Specifically,  the  mission,  function  and  task  analyses  will  drive  the  process  in  the  short 
term. 

Additional  items  that  should  be  in  the  reports  that  you  expected  to  find. 

Risk  analysis  and  cost/benefits  should  be  provided  for  the  major  HF  tasks  in  order  to  allow  the  PM 
to  make  the  necessary  trade-offs. 

More  detailed  information  should  be  provided  on  the  relationship  between  the  HF  analyses  and 
system  requirements  specification. 


3.5.3.  Summary  of  interview  with  Project  Manager  ARDS/ADM 
Background 

The  PM  described  himself  as  being  originally  an  end-user  artillery  officer  who  was  promoted  into 
procurement  and  project  management,  and  has  been  involved  with  C2  issues  since  1988. 

He  was  previously  familiar  with  the  MANPRINT  program  and  saw  some  problems  with  the  initial 
plan  for  ARDS/ADM  development. 

Report  style  and  organisation 

The  writing  flowed  well.  It  was  easy  to  understand  because  it  follows  the  operational  planning 
process  closely;  therefore  an  army  person  would  relate  to  it.  The  reports  were  written  at  an 
appropriate  level  of  technical  language  that  military  users  would  understand. 

HFE  Process  and  method 

The  process  and  method  were  adequately  described. 

The  major  difficulty  is  in  the  lack  of  explanation  as  to  how  the  HEPP  fits  in  with  the  overall 
systems  engineering  planning  approach.  This  relationship  should  be  made  explicit  with  a 
schematic  model.  The  reports  read  as  if  the  HF  plan  drives  the  systems  engineering  process, 
instead  of  being  integrated  with  it  (particularly  in  areas  such  as  mission  and  function  analysis). 
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The  reports  tell  you  what  to  do  but  are  not  clear  on  when  to  do  it  or  how  to  integrate  with  other 
processes. 

The  requirements  development  process  needs  to  be  better  tied  to  the  systems  engineering  plan. 

Newer  approaches  to  project  management  and  systems  engineering,  such  as  quality  function 
deployment  (QFD),  will  provide  as  yet  unsolved  challenges  on  how  to  integrate  HFE  and  how  to 
identify  and  evaluate  the  HF  component  in  a  proposal  that  uses  a  QFD  approach. 

How  the  information  contained  in  the  reports  might  have  influenced  the  ARDS/ADM 
approach. 

If  the  reports  had  been  available  at  the  start  of  the  project,  it  would  have  been  of  considerable 
benefit  for  planning  and  prioritising  the  initial  major  HF  tasks  to  be  done. 

The  reports  make  a  sound  case  for  the  early  identification  of  deficiencies  and  provide  an  excellent 
account  of  the  process  to  be  followed  for  such  identification. 

The  section  dealing  with  capability  deficiencies  would  have  been  useful  to  formalise  a  method. 

The  reports  would  make  it  easier  to  assign  priorities  and  make  early  decisions.  The  proposed 
approach  forces  you  to  look  at  entire  requirement. 

The  proposed  approach  allows  the  clients  to  do  their  homework  and  define  the  operational  concept 
early.  The  mission/function  analysis  leads  naturally  to  understanding  the  required  operational 
architecture. 

The  reports  lay  out  a  formal  process.  This  makes  for  generic  consistent  statements  and  the  process 
of  arriving  at  them  is  accountable  and  transparent. 

The  User  Manual  clearly  identifies  the  role  of  client  and  level  of  effort  required  in  order  to  properly 
define  requirements. 

Additional  items  that  should  be  in  the  reports  that  you  expected  to  find. 

The  reports  do  not  provide  appropriate  information  on  where  to  make  HF  trade-offs. 

The  business  case  for  conducting  a  full  HF  program  needs  to  be  spelled  out.  A  cost/benefit 
analysis  would  be  useful.  More  information  concerning  the  actual  estimates  of  time  and  cost 
should  be  provided  together  with  the  expected  benefits/risks. 

More  detail  should  be  provided  on  the  scope  and  costs  of  the  major  HF  activities  and  also  their 
timeline  and  logistics. 

The  general  systems  requirements  for  function  allocation  are  not  dealt  with  adequately.  The  focus 
of  energies  cannot  just  be  on  either  the  human  or  the  hardware  side.  Also,  there  should  be  a  section 
that  deals  with  function  allocation  among  personnel. 

There  should  be  a  section  on  how  IV&V  fits  in  with  the  overall  process. 

The  reports  should  provide  more  guidance  on  how  to  move  from  the  HF  analysis  to  the  statement 
of  requirements. 
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3.5.4.  Summary  of  interview  with  Program  Manager/Government  Focal  Point 
Displays  and  for  HFE  for  the  Internal  Communication  System  R/SAOC 
Modernisation  Program 

Background 

The  PM  had  no  previous  specific  knowledge  concerning  HFE,  just  that  acquired  as  part  of  an 
aerospace  systems  course.  He  had  gathered  expertise  as  the  project  evolved  from  DCIEM,  who 
provided  mostly  verbal  guidance.  He  was  familiar  with  the  HSI  report  on  MOPs  and  Framework 
for  Evaluation. 

Report  style  and  organisation 

The  reports  are  not  as  “user  friendly”  as  they  could  be  and  read  like  a  textbook.  The  content  can  be 
quite  verbose,  contains  repeated  material  and  places  too  many  demands  on  memory  across  the 
various  sections. 

The  User  Manual  was  really  liked,  but  there  was  less  enthusiasm  for  the  Guidelines  report,  and  the 
sections  on  Requirements  were  found  to  be  quite  challenging,  even  with  a  background  in  the 
material. 

The  reports  make  demands  on  the  reader  to  remember  too  much  material  across  the  two  reports. 
There  is  a  need  for  a  better  introduction  and  lead-in  to  many  sections 

Unless  it  was  an  officer’s  job  to  read  them,  the  reports  would  be  discarded  and  put  down  quickly. 
This  is  because  the  information  is  not  presented  and  organised  in  a  manner  that  makes  it  user 
friendly. 

More  military  examples  should  be  provided  in  the  User  Manual  both  for  comprehension  and 
making  the  business  case. 

HFE  Process  and  method 

The  relationship  between  the  HF  plan  and  other  approaches  to  the  systems  engineering  process 
needs  to  be  explained.  The  proposed  approach  does  not  fit  the  R/SAOC  IPT  (integrated  program 
teams)  model  well,  this  is  a  core  part  of  IPPD  process  that  has  been  adopted  by  US  Department  of 
Defence.  The  Canadian  Air  Force  is  moving  towards  adopting  this  approach  and  the  R/SAOC 
program  was  used  as  a  model. 

The  reports  are  weak  in  describing  the  role  of  IV&V. 

The  reports  do  not  show  clearly  the  chain  of  command  in  the  HF  team  in  project  management 
hierarchy.  There  is  a  need  to  spell  out  reporting  relationships  and  to  encourage  the  HF  plan 
management  to  be  high  in  the  program  management  hierarchy. 

The  reports  are  very  strong  on  the  pre-contractual  work  that  needs  to  be  done. 

The  reports  should  emphasise  the  importance  of  creating  a  database  to  track  the  HF  plan  (e.g. 
usability  trial  comments,  IPT  notes,  who  said  what  and  what  was  done).  The  adoption  of  a 
usability  tracking  matrix  was  suggested  in  order  to  maintain  a  log  of  when  each  issue  is  to  be 
addressed  (which  build  and  evaluation  trial  in  an  iterative  process),  the  priority  of  each  issue  and 
the  outcome. 
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How  the  information  contained  in  the  report  might  have  influenced  the  R/SAOC 
program  development 

The  reports  are  very  useful  and  it  would  have  been  highly  desirable  to  have  had  these  available  at 
the  start  of  the  project.  It  would  have  guided  much  of  the  early  processes  and  would  have  spelled 
out  clearly  the  need  for  the  contractor  delivering,  and  committing  to,  a  viable  HEPP. 

The  sections  on  developing  statements  of  deficiencies  are  potentially  very  useful  and  would  have 
resulted  in  a  much  clearer  statement  of  deficiencies  (note:  he  has  never  been  on  a  project  that  starts 
with  deficiency  requirements).  Other  sections  that  would  have  influenced  the  evolution  of  the 
project  are  the  statements  on  the  HEPP  deliverables  and  making  the  case  for  the  HF  effort. 

The  reports  would  have  helped  to  justify  the  HFE  budget  and  plan  budget  better. 

The  reports  would  have  been  of  benefit  to  the  project  office  in  planning  and  integration  the  HF 
effort  throughout  the  design,  development,  fielding  cycle  by  providing  clear  guideline  on  the 
process. 

Additional  items  that  should  be  in  the  reports  that  you  expected  to  find. 

The  section  on  budget  is  useful  but  should  be  enhanced  with  some  detailed  budget  examples  drawn 
from  previous  projects  in  order  to  enhance  the  authority  of  the  document.  These  examples  should 
deal  with  making  the  business  case  for  HF  and  provide  an  indication  of  the  cost  payback  for  the 
investment. 

There  is  a  need  to  develop  a  generic  “briefing  package”  that  could  be  used  to  make  the  case  for  the 
HF  program. 

There  should  be  a  section  on  the  risks/benefits  of  the  various  HF  tasks. 

There  should  be  a  section  outlining  the  need  for  early  identification  of  the  software  required  for  HF 
test  and  evaluation,  since  the  specific  design  for  this  software  will  have  to  be  integrated  into  the 
overall  systems  software  program.  (Note:  a  similar  case  can  be  made  for  the  development  of 
system  maintenance  software). 
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4.  Recommendations  for  issues  to  be 
considered  by  the  TBCS  project  office 


Any  detailed  discussion  of  specific  steps  to  be  taken  by  the  TBCS  project  office,  as  guided  by  the 
Engel  and  Townsend  reports  is  moot,  given  that  the  TBCS  project  office  has  made  the  decision  to 
conduct  a  mission,  task  and  function  analysis  as  the  next  step  in  system  development.  In  general,  we 
believe  that  based  upon  the  overall  HEPP  report  and  our  experience  with  the  TBCS  project  to  date,  the 
mission  and  task  analysis  is  an  appropriate  step  to  be  accomplished  before  moving  forward  in  the 
system  design  and  development  process.  Notwithstanding  the  scope  of  the  HEPP  outlined  in  the 
Guidelines  report  and  TBCS  SOW,  we  make  some  recommendations  in  this  section  for  maximising 
the  benefit  to  be  gained  from  the  ongoing  analysis.  Attention  to  the  issues  outlined  below  may  mean  a 
more  extended  analysis  will  need  to  be  performed  in  some  specific  areas.  As  such,  these 
recommendations  go  beyond  the  existing  HEPP  Guidelines  report  and  existing  TBCS  SOW  for  the 
next  phase  of  the  project. 

4.1.  Nature  of  the  human  factors  data  to  be  generated 

It  is  important  at  the  outset  of  the  next  stage  of  work  that  the  form  and  nature  of  the  data  obtained 
from  the  mission,  function  and  task  analysis  are  clearly  understood  with  respect  to  the  ability  of  the 
data  to  support  subsequent  requirements  definition  and  design  guidance.  That  is,  the  raw 
descriptive  data  that  can  be  a  typical  product  of  such  an  analysis  may  be  insufficient  to  inform  this 
process.  Interviewee  #4,  made  a  strong  point  of  this  in  the  context  of  the  difficulty  that  systems 
developers  had  in  understanding  the  workload  analysis  data.  Thus,  one  of  the  primary  goals  in 
going  into  the  analysis  will  be  to  arrive  at  an  understanding  of  what  forms  of  data  will  be  useful  for 
driving  subsequent  requirements  specification  and  design  decisions,  and  what  further  analyses  of 
the  descriptive  data  or  tasks  may  be  required  to  generate  such  data. 

A  further  decision  to  be  made  in  advance  will  be  to  define  what  constitutes  critical  information  in 
the  analyses  and  how  this  can  be  recognised  and  extracted  from  the  large  amount  of  data  generated. 
For  example,  criteria  will  need  to  be  established  to  determine  how  the  workload  metrics  generated 
by  the  analysis  can  be  interpreted  in  system  design  terms.  Traditionally,  workload  metrics  have 
been  used  for  comparative  assessments  of  different  system  states.  However,  for  efficient  design 
guidance,  HFE  specialists  will  need  to  provide  recommendations  to  system  designers  on  what 
values  of  workload  are  salient  when  they  review  specific  areas  of  system  functionality  in  order  to 
reduce  potential  overload.  These  numbers  should  be  precise  enough  to  identify  those  critical  C2IS 
processes  that  will  demand  particular  attention  in  terms  of  design  support  and  requirements 
specification.  The  alternate  approach  of  running  multiple  simulations,  while  manipulating  system 
parameters  to  achieve  the  required  reduction  in  the  modelled  workload  values,  is  probably  not  cost 
effective  at  this  early  stage  of  system  analysis.  Further,  without  specific  HFE  analysis  to  guide  the 
necessary  revisions  to  the  modelled  system,  arriving  at  an  optimum  or  satisfactory  solution  may  be 
a  hit  or  miss  process.  It  will  also  be  necessary  to  specify  in  advance  approximate  workload  metric 
values  for  effective  system  performance,  in  order  to  know  when  a  satisfactory  modelled  solution  is 
appropriate  and  acceptable. 

One  further  issue  concerning  the  HF  data  generated  is  the  extent  to  which  the  future  system  designs 
are  expected  to  be  evolutionary  or  revolutionary.  In  the  case  of  the  former,  one  would  expect  a 
high  degree  of  utility  in  applying  HFE  analyses  based  existing  C2IS  systems  and  processes. 
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However,  if  future  systems  are  expected  to  involve  a  major  rethinking  in  the  way  C2IS  processes 
could  be  implemented  more  effectively  with  newly  emerging  technologies,  then  the  transfer  and 
applicability  HFE  data  derived  from  task  analysis  of  existing  systems  has  the  potential  to  be 
uninformative  to  those  responsible  for  shaping  systems  requirements,  and,  ultimately,  system 
designers.  Instead,  what  will  be  of  value  is  a  high  level  function  analysis  of  the  overall  mission 
objectives  and  processes  that  will  need  to  be  performed,  no  matter  what  the  ultimate  shape  of  the 
final  technology. 

4.2.  Analysis  of  situation  awareness  requirements 

It  remains  to  be  seen  whether  the  forms  of  analysis  proposed  by  the  TBCS  PMO  in  the  SOW  for  the 
next  step  in  TBCS  development  can  address  the  detailed  requirements  specification  and  decisions 
that  will  need  to  be  made  concerning  system  design  to  support  situation  awareness.  The  latter  has 
been  established  as  one  of  the  key  factors  underlying  effective  C2  performance  and  confirmed  in  the 
preliminary  cognitive  analysis  of  combat  team  functions  performed  by  HSI.  Interviewee  #4 
identified  the  importance  and  usefulness  of  conducting  such  an  analysis  (as  a  supplement  to 
workload  analysis)  in  providing  a  basis  for  detailed  design  decisions.  Whatever  the  form  of  the 
analysis  ultimately  adopted,  the  ensuing  data  should  inform  the  PMO  very  precisely  about  those 
mission  functions  that  depend  critically  on  good  situation  awareness.  In  particular,  these  mission 
functions  will  need  to  be  analysed  from  the  perspective  of  the  information  requirements  to  support 
all  three  levels  of  situation  awareness  (reference  1 1).  While  the  SOW  addresses  the  need  to  identify 
the  information  requirements  in  order  to  perform  tasks,  and  the  need  to  define  the  cognitive 
components  of  the  task,  the  SOW  fails  to  specify  how  these  two  elements  interact  to  shape,  for 
example,  the  specific  user’s  information  needs.  Defining  the  information  required  to  conduct  a  task 
and  the  cognitive  process  involved,  in  itself  tells  nothing  about  the  way  to  optimise  the  system 
design  to  enhance  information  processing.  Analysis  of  situation  awareness  requirements  would  be 
one  area  that  would  benefit  from  the  more  detailed  analysis  of  this  complex  interaction. 

4.3.  Analysis  of  communication  requirements 

Communication  effectiveness  has  been  identified  as  being  a  core  requirement  for  the  successful 
achievement  many  C2  mission  goals,  tasks  and  sub-tasks.  The  HFE  approach  to  dealing  with 
communication  issues  outlined  in  the  reports,  and  the  TBCS  SOW,  deals  with  communication  flow 
primarily  through  analyses  based  upon  operational  sequence  diagrams.  For  such  analyses  to  be 
useful  from  a  system  specification  and  design  perspective  they  will  need  to  go  beyond  identifying 
message  sources,  destinations,  flow,  traffic  peaks,  overlaps  and  lags.  The  analyses  will  need  to  take 
into  account  the  cognitive  requirements  for  the  transmitted  information  in  terms  of  many  factors 
relating  to  how  the  transmitted  information  influences  situation  awareness  and  decision  making. 
Message  data  in  themselves  only  become  meaningful  to  the  recipient  in  the  context  of  these  two 
factors.  Issue  such  as  message  salience,  immediacy,  timeliness,  reliability,  uncertainty  must  all  be 
considered  in  the  context  of  understanding  message  information.  In  order  to  meet  the  needs  for 
specifying  system  design,  these  issues  will  need  to  be  addressed  by  extending  the  analysis  to  yield 
relevant  data  that  go  beyond  the  typical  descriptive  flow  diagrams  produced  in  operational  sequence 
analysis.  Further  consideration  also  needs  to  be  given  to  the  management  of  communications  in  the 
high  volume  environment  that  is  typical  of  many  military  applications. 
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4.4.  Integration  with  other  systems,  scope  and  boundary  issues 

Clearly  the  analysis  for  TBCS  C2IS  needs  must  take  into  account  the  broader  command  and 
communication  infrastructure  and  other  military  engineering  systems  within  which  TBCS  will 
operate.  The  flow  of  information  down  from  LFIS  through  TBCS  to  IPCE  will  require  not  only  an 
HFE  analysis  within  the  combat  team,  but  also  at  the  critical  points  of  interface  with  other  C2 
systems.  There  are  no  specific,  detailed  HEPP  guidelines  as  to  how  this  analysis  is  to  be  conducted 
and  some  initial  planning  will  be  required  to  understand  just  what  can  and  cannot  be  achieved  with 
the  standard  HEPP  HFE  analyses  proposed. 

A  further  factor  to  be  considered  will  be  the  potential  conflict  between  system  wide  information 
formatting  demands  and  local  requirements.  On  the  one  hand,  there  will  be  design  constraints  to 
ensure  consistency  of  data  formats  between  LFIS/TBCS  and  TBCS/IPCE  to  avoid  confusions 
among  users  who  may  operate  at  the  boundary  points  in  order  to  perform  mission  tasks  (or  whose 
training  is  predominately  with  one  system  or  the  other).  On  the  other  hand,  the  need  for  these 
constraints  will  need  to  be  balanced  against  the  appropriateness  of  the  system  design  when 
optimised  for  local  TBCS  requirements. 

Similarly,  it  is  likely  that  within  the  combat  vehicle,  the  TBCS  will  interface  with  weapons  and 
potentially  vehicle  mechanical  and  operational  systems.  Hence  there  will  be  a  need  for  detailed 
analyses  to  understand  the  user’s  information  requirements  and  information  flow  across  and 
between  systems. 

4.5.  Doctrinal  constraints 

In  considering  the  scope  and  goals  of  the  function  analysis,  early  decisions  will  need  to  be  made 
concerning  the  degree  to  which  a  “green  fields”  approach  will  be  taken,  that  is,  there  are  no  prior 
constraints  to  thinking  about  possible  function  allocations.  This  would  mean  that  existing  doctrinal 
issues  concerning  the  operational  missions  performed  by  combat  teams  and  the  individual  roles 
performed  by  team  members  will  not  be  relevant  factors  in  optimising  the  allocation  of  functions. 
However,  this  is  unlikely  to  be  a  realistic  approach  and  hence  there  are  likely  to  be  many 
constraints  involving  these  factors.  This  will  certainly  reduce  the  scope  of  the  function  analyses  if 
there  are  early  decisions  to  respect  existing  doctrinal  constraints.  If  this  is  the  case,  much  of  the 
functional  allocation  issues  will  be  centred  on  human/system  trade-offs. 

4.6.  Over-analysis 

One  of  the  dangers  in  a  major  mission,  function  and  task  analysis  is  the  danger  to  over-analyse  the 
obvious  or  simple.  Informed  HFE  analysis  involves  not  only  the  use  of  computer  based  analytical 
techniques,  but  also  expert  analyses  by  experienced  HF  specialists  together  with  the  appropriate 
army  SMEs.  The  issue  here  is  often  one  of  cost  effectiveness  in  terms  of  finding  the  fastest  and 
most  efficient  way  to  answer  a  design  or  requirements  definition  issue.  For  example,  given  the 
well  documented  high  error  rates  associated  with  the  transcription  process  of  plotting  map-based 
information  from  text  or  audio  sources  (whether  in  a  planning  phase,  or  under  actual  battle 
conditions),  the  ability  to  automatically  send  graphic  overlays,  traces  or  symbols  with  correctly 
plotted  data  to  update  the  users  current  map,  would  be  of  considerable  benefit.  While  it  is  true  that 
all  of  the  individual  tasks  and  actions  which  underlie  this  processes  of  passing  spatial  information 
could  be  readily  modelled,  it  is  not  clear  that  having  a  “precise”  number  to  reflect  the  saving  in 
time  or  errors  would  enhance  the  design  decision  reached. 
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There  remains  a  danger  to  the  credibility  of  the  profession  if  we  can  provide  only  expensive, 
lengthy  and  minutely  crafted  answers  to  questions  that  can  be  addressed  with  relative  ease  and 
efficiency  by  less  involved  processes. 

Some  guidelines  on  the  selection  of  appropriate  HFE  analysis  for  systems  of  varying  complexity 
can  be  found  in  recommendations  provided  by  NATO  DRG  Panel-8  RSG.14  (reference  9). 


4.7.  Team  requirements 

Issues  relating  to  the  effective  performance  of  teams  have  emerged  only  recently  in  the  HFE 
literature.  Consequently,  many  of  the  traditional  analytical  methodologies  have  yet  to  incorporate 
sections  into  a  HEPP  to  deal  with  team  performance  issues.  While  it  is  beyond  the  scope  of  the 
present  report  to  provide  details  on  how  to  incorporate  analysis  for  team  performance  into  a  HEPP, 
this  issue  is  “flagged”  as  one  that  will  require  early  attention  as  TBCS  moves  into  the  next  phase  of 
the  system  analysis. 

It  is  possible  that  all  of  the  team  issues  can  be  accommodated  within  the  planned  analyses,  however 
early  cognisance  of  issues  which  go  beyond  the  individual  performance  of  team  members  may 
need  to  be  addressed.  Some  of  these  include  team  situation  awareness,  “shadowing”  by  some  team 
members  of  the  functions  performed  by  others,  implicit  knowledge  that  exists  only  at  the  team 
level  and  team  decision  making.  Observations  of  “team  behaviours”  during  simulations  and 
exercises  suggest  that  factors  such  as  “body  language”,  voice  intonation,  gestures  and  facial 
expressions  all  carry  important  cues  for  communicating  aspects  of  common  intent.  To  date, 
traditional  approaches  to  task  analyses  have  not  tended  to  capture  such  information. 
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5.  Recommendations  for  next  steps  for 
developing  the  HEPP  reports. 


In  this  section  we  make  some  suggestions  on  how  DCIEM  might  go  beyond  the  existing  material 
provided  by  Engel  and  Townsend  in  order  to  provide  military  users  with  a  comprehensive 
reference  package  that  would  provide  assistance  in  the  integration  of  HFE  into  the  systems 
development  process.  These  suggestions  are  based  partly  on  our  own  ideas,  augmented  and 
reinforced  by  comments  and  suggestions  arising  from  the  interviews. 

The  Engel  and  Townsend  reports  provide  the  majority  of  the  core  content  that  is  required  for  a 
comprehensive  HEPP  for  CCIS  development  and  this  will  create  a  sound  foundation  for  any  future 
work.  The  suggestions  provided  below  for  expanding  and  improving  the  potential  utility  and 
usability  of  the  document  and  may  be  categorised  into  four  main  areas:  (i)  expanding  the  core 
content,  (ii)  enhancing  utility,  (iii)  enhancing  usability  and  (iv)  integration  with  other  relevant 
material.  Within  the  limitations  of  the  scope  of  the  present  work  and  budget,  these  issues  will  be 
described  in  point  form  only. 

5.1 .  Expand  the  core  content 

•  Broaden  the  scope  of  task  and  mission  analyses  to  include  additional  core  concepts  derived 
from  NATO  STANAG  3994  and  other  relevant  documents. 

•  Integrate  and  expand  issues  relating  to  measures  of  performance  and  effectiveness  for 
CCIS  systems. 

•  Tailor  the  section  on  decision  support  requirements  to  the  specific  context  of  a  CCIS. 

•  Expand  the  section  on  HF  analyses  to  incorporate  the  other  core  processes  that  support 
effect  C2IS,  (situation  awareness,  communication  effectiveness  and  decision  making,  in 
addition  to  workload)  and  include  unique  HF  issues  related  to  team  functions. 

•  Provide  additional  information  on  the  important  role  to  be  played  by  HFE  specialists  in 
translating  data  from  HFE  analyses  into  design  requirements  and  recommendations. 

5.2.  Enhance  utility 

•  Provide  a  section  that  outlines  the  “business  case”  for  the  HFE  effort;  consider  the 
provision  of  a  “briefing  package”  that  PMs  could  use  to  support  arguments  to  superiors 
about  the  need  for  a  strong  HFE  program. 

•  Provide  a  section  that  shows  the  costs  and  benefits  of  the  major  HFE  activities.  In 
particular,  document  the  risks  associated  with  not  performing  specific  analyses. 

•  Show  how  the  HEPP  integrates  with  the  overall  systems  engineering  plan.  In  particular, 
consider  the  specific  challenges  associated  with  locating  the  HFE  effort  in  emerging 
systems  engineering  processes. 

•  Clearly  differentiate  the  HF  responsibilities  and  activities  done  by  the  PM  team  prior  to 
contract  award  and  throughout  the  system  development  process,  from  those  performed  by 
the  contractor  and  the  IV&Y  team. 
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•  Provide  more  specific  guidance  on  how  the  data  that  are  output  from  HF  analyses  can  be 
used  to  inform  initially  the  process  of  requirements  development,  and  later,  the  detailed 
system  design. 

•  Broaden  the  scope  to  include  issues  that  may  be  unique  to  the  Navy  and  Air  Force 
communities. 

5.3.  Enhance  usability 

•  At  present  the  surface  structure  (i.e.  the  interface)  of  the  reports  is  not  conducive  to  rapid 
acquisition  or  comprehension  of  the  information,  particularly  for  the  brief  time  slices  that 
busy  military  personnel  seem  able  to  allocate  to  reading  such  material.  There  are  large 
sections  of  unbroken,  single  font,  single  pitch  text,  and  reviewers  commented  that  the 
document  read  like  a  textbook.  This  surface  structure  could  be  improved  by  the  adoption 
of  an  overall  document  style  that  makes  fewer  demands  on  the  reader.  A  number  of  HF 
and  document  design  guides  are  available  to  provide  appropriate  style  recommendations. 
More  graphics  and  tables  should  also  be  provided  to  amplify  and  illustrate  main  points. 

•  Provide  a  means  by  which  users  can  quickly  locate  topics  of  interest. 

•  Consider  developing  a  hypertext  document  to  provide  improved  navigation  within  and 
between  the  reports  and  links  between  related  topics. 

•  Wherever  possible  provide  strong,  illustrative,  military  examples  to  facilitate 
comprehension  of  major  HF  analyses,  methods  and  data.  One  possibility  would  be  to 
produce  a  CD-ROM  or  video  with  example  case  studies  that  depict  difficult  decisions  and 
problems  facing  the  PM,  and  appropriate  steps  to  be  taken. 

•  Several  military  officers  have  suggested  that  paper  as  a  medium  is  going  out  of  favour  at 
DND  and  that  the  primary  means  of  information  transmission  is  electronic.  This  results  in  a 
reduced  willingness  to  spend  time  with  paper  as  a  medium  and  to  read  lengthy  paper-based 
documents.  Therefore,  some  consideration  should  be  given  to  producing  an  electronic 
document  for  military  use.  This  should  go  beyond  the  simple  conversion  of  paper  into 
diskette,  but  should  take  advantage  of  the  additional  learning  benefits  that  accrue  with  well- 
designed,  multimedia  information  packages.  Thus,  many  of  the  above  recommendations 
could  be  accommodated  in  a  highly  effective  manner  through  the  production  of  an 
interactive  CD.  This  would  supplement  the  existing  core  material  with  a  more  effective 
interface,  improved  means  of  rapid  access  to  critical  information,  improved  linkage  between 
related  issues,  improved  graphics,  specific  examples  to  illustrate  points  etc. 

5.4.  Integrate  with  other  relevant  documents 

•  DCIEM  has  itself  produced  many  reports  on  applying  HFE  to  system  design  and  it  has  also 
commissioned  contractors  to  provide  HFE  analyses  of  military  systems.  Much  of  this 
information  is  highly  relevant  to  the  present  context  and  has  the  potential  to  fill  some  of  the 
present  gaps  in  coverage.  Therefore,  means  should  be  found  of  supplementing  the  core 
content  of  the  present  reports  with  references  to  relevant  material  in  these  other  documents. 

•  Provide  references  to  other  related  non-DCIEM  material,  e.g.  NATO  STANAG,  recent  US 
technical  reports. 
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Annex  A:  Interview  Data 


A.1  Point  form  transcription  of  interview  with  the  Leader,  Functions  Group, 
Information  Systems  Technology  Group  DREV  at  DREV,  July  6  1998. 


Background  knowledge 

Have  you  had  an  opportunity  to  read  and/or  study  any  similar  or  related  documents  previously  -  if 
so  -  what  were  they ? 

Has  read  some  DND  technical  reports  on  some  on  requirements  capturing  but  nothing  on  HFE. 

Has  some  familiarity  with  HCI  guidelines  (e.g.  Engel  and  Townsend  report). 


Report  organisation  and  style 

Was  the  report  organised  in  manner  that  made  sensei 

Yes  -  in  particular  liked  summaries.  It  is  dense-more  examples  -a  PM  will  not  take  time  to  go 
through.  Need  some  up  front  buy  in.  Need  more  examples.  Also  need  to  make  surface  structure 
user  friendly.  Use  consistent  example  throughout  e.g.  planning. 


Was  the  report  organised  in  manner  that  enabled  you  to  understand  the  various  HEPP  components 
and  how  they  related  to  one  another? 

Yes- very  clear. 


Was  the  report  written  in  the  appropriate  level  of  technical  language? 

The  writing  was  dense  and  concentrated,  but  not  too  technical.  Liked  descriptions  of  mock-up, 
prototyping  environment. 

Do  you  think  that  the  typical  military  officer  would  have  difficulty  in  understanding  the  technical 
aspects  of  the  report? 

Users  probably  will  have  difficulty  in  reading  all  of  the  technical  aspects  of  the  report  unless  the 
reader  can  be  convinced  initially  of  the  importance  of  HFE  to  the  overall  systems  engineering 
effort.  The  user  needs  clear  up  front  motivation  to  keep  going  through  the  detailed  technical 
sections. 

There  needs  to  be  a  greater  number  of  examples  to  illustrate  the  text,  particularly  for  task  and 
function  analysis.  There  is  no  guidance  as  to  exactly  what  to  do  with  the  outcome  of  the  analyses. 
There  is  not  enough  information  to  allow  a  PM  to  recognise  a  good  task  analysis  from  a  bad  one. 

The  examples  provided  for  formulating  statements  of  deficiencies  were  found  to  be  very  useful. 

The  detailed  sections  on  tailoring  the  HFE  plan  (5 .3 .2.2  and  subsequent)  are  hard  to  follow  because 
of  technical  density,  and  again  examples  would  help  in  comprehension. 

More  examples  should  be  provided  to  illustrate  DIDS. 
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Was  it  useful  to  divide  the  report  into  separate  guidelines  and  implementation  sections? 

Yes.  However,  if  users  read  only  the  “guidelines”  then  they  will  need  a  better  introductory 
summary  on  what  makes  a  good  HFE. 

May  also  need  to  provide  greater  rationale  and  incentive  to  convince  a  contractor  to  implement  the 
full  HFE  plan  (if  only  the  “Guidelines”  document  were  to  be  given  to  them). 

Generally  believed  that  army  users  would  find  the  first  part  of  the  user  manual  (not  detailed 
description  of  DIDS)  more  useful,  supplemented  by  the  summary  in  the  “Guidelines”  section. 


Report  contents  and  concepts 

Were  there  any  new  concepts  in  the  report  which  you  had  not  encountered  previously? 

Generally  not  familiar  with  full  scope  of  HFE,  in  particular  what  was  involved  in  task  and  function 
analysis  (only  familiar  with  HCI  issues). 

Need  more  examples  -since  it  was  not  easy  to  understand  all  concepts. 

Suggestion  is  to  follow  one  example  throughout,  e.g.  the  flow  of  an  Op  Order.  This  could  then  be 
decomposed  using  task  analysis  and  OSDs,  storyboard  of  design  prototypes. 

It  was  not  clear  how  OSDs  relate  to  regular  data  and  process  models.  Will  HFE  methods  allow 
everything  to  be  represented  in  the  data  model?  Seriously  questions  whether  this  can  this  be  done 
for  emerging  technologies  without  a  physical  or  some  form  of  prototyping  mock-up.  Sees  the  use 
of  mock-ups  to  assist  task  decomposition. 

The  section  on  Hartson  and  Hix  should  be  expanded.  (Note:  this  approach  was  strongly  endorsed). 

Would  like  to  see  a  greater  emphasis  on  iterative  development  as  opposed  to  the 
evolutionaiy/waterfall  methods  proposed. 

Does  not  buy  into  the  differences  outlined  in  the  report  in  the  requirements  approach  to  the 
development  of  CCIS  versus  hardware.  While  a  distinction  may  exist  now,  does  not  believe  that 
this  is  the  way  it  should  be,  since  there  should  be  a  human  should  be  focus  for  both  types  of  system 
in  the  future.  This  is  where  simulation  based  acquisition  methods  are  particularly  useful  for 
hardware  acquisition. 

Advocates  a  hybrid  approach  that  combines  prototyping  with  more  formal  HFE  methods. 


Were  there  any  concepts  with  which  you  were  previously  familiar,  but  became  clearer  to  you  as  a 
result  of  reading  the  reports ? 

Yes  for  task  analysis  and  HFE  scope  in  general.  Scope  and  cost  information  was  new;  believes  that 
the  reports  underestimate  the  true  cost  of  HFE  in  iterative  development. 

The  reports  are  not  clear  or  detailed  enough  on  how  to  integrate  HFE  with  other  systems 
engineering  activities.  The  reports  seem  to  be  strongly  pushing  the  insertion  of  a  new  “block”  into 
normal  systems  engineering  process  approach.  This  may  be  perceived  as  threatening  to  systems 
engineers  and  PM’s.  The  reports  are  seen  as  being  pushed  the  “independent  HFE  model”  too  hard. 

Believes  that  HFE  is  best  integrated  during  the  development  of  functional  requirements.  Whereas 
the  reports  seem  to  imply  HFE  analysis  is  done  beforehand,  instead  of  in  parallel  with  other 
technical  analysis.  Further,  it  is  not  obvious  how  HFE  is  integrated  into  life  cycle  systems 
development. 

Believes  that  you  cannot  write  HFE  deficiencies  without  knowing  what  technologies  are  feasible. 
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Agrees  that  5-10%  estimate  of  total  project  cost  for  the  HFE  effort  is  reasonable,  but  taking  into 
account  all  of  the  required  documentation,  believes  that  overall  it  would  account  for  more  than 
10%.  “Chameleon  provides  the  living  documentation.” 

Would  like  to  have  seen  how  HFE  activities  interact  with  ongoing  prototype  development;  does  not 
believe  that  you  can  first  do  task  analysis,  then  validate  and  then  design. 


Were  there  concepts  you  would  have  expected  to  find  in  the  reports  but  were  missing? 

A  greater  description  of  the  Hartson  and  Hix  approach  (because  this  is  seen  as  a  viable  approach.) 


More  detailed  questions 

Which  aspects  of  the  reports  would  have  been  useful  at  the  very  outset  of  the  TBCS  project? 

The  lessons  learned  from  the  TBCS  field  trial  provided  a  context  for  understanding  how  some  of 
the  aspects  of  the  reports  actually  can  be  applied,  in  particular,  the  need  for  early  mission  and  task 
analysis. 

Would  you  have  done  anything  differently  at  the  outset  of  the  project  had  the  reports  been 
available? 

Yes.  Instead  of  going  to  the  design  of  concept  prototypes  would  have  done  more  initial  analysis. 
They  would  have  adopted  iterative  approach  of  “analyse  a  bit,  design  and  develop  a  bit,  test  a  bit”. 

Would  the  reports  have  changed  the  approach  adopted  by  DREV  and  TBCS,  namely,  moving 
quickly  forward  with  rapid-prototyping  and  producing  Chameleon? 

No  -  still  need  to  demonstrate  prototype  and  validate  through  a  scenario  prior  to  more  substantive 
HFE  analysis  activities  such  as  function  allocation  and  OSD’s.  However,  a  prior  mission  analysis 
would  allow  priorities  for  the  first  prototype  to  be  more  easily  identified. 


Given  the  current  stage  of  development  of  the  TBCS  project,  do  the  reports  provide  insight  into 
incorporating  HF  aspects  into  system  design  in  the  next  phases  of  the  project? 

Yes,  first  step  back  and  do  an  appropriate  mission  analysis  and  task  analysis.  Follow  this  with  a 
review  of  function  allocation  and  compare  the  product  with  Chameleon  prototypes  to  date. 

The  test  scenario  demonstration  at  Pettawawa  was  very  useful  for  identifying  Chameleon 
deficiencies.  This  demonstrates  that  if  HFE  had  been  there  from  the  start  it  would  have  been 
integrated  into  development  cycle  and  the  current  step  back  would  be  necessary. 


Does  it  provide  guidance  on  what  are  the  next  most  important  HF  considerations?  How  does  it  do 
that ? 

Yes  -  will  need  to  ensure  that  the  next  implementation  reflects  all  HFE  considerations,  in  particular 
the  need  for  formal  testing.  The  reports  suggest  that  a  next  obvious  step  would  be  to  generate  a 
special  set  of  HFE  requirements  as  part  of  functional  SOR.  There  will  also  be  a  need  to  integrate 
HFE  into  predefinition  activities,  for  storyboarding  concepts,  technology  demos,  or  simulation 
based  acquisition. 
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Do  the  reports  provide  insight  into  the  various  HF  responsibilities  throughout  the  project 
acquisition  cycle? 

No  -reports  imply  just  roles  for  contractor,  PM  and  users,  what  about  SMEs?  The  reports  do  not 
spell  out  the  precise  make  up  of  HFE  team  and  individual  responsibilities. 

Would  the  reports  assist  the  project  office  in  planning  and  integration  the  HF  effort  throughout  the 
design,  development,  fielding  cycle? 

The  reports  need  to  show  how  the  HFE  activities  are  better  fused  throughout  the 
design/development  cycle.  It  implies  separate  but  connected  processes-but  would  like  to  see  a 
more  integrated  approach.  E.g.  in  ARDS/ADM  the  analysis  was  useful  but  inadequately 
incorporated  into  the  design  of  a  new  automated  system. 


Do  the  reports  provide  you  with  sufficient  information  to  understand  the  benefits  of  each  of  the 
major  HF  activities  ?  Yes. 


Do  the  reports  provide  you  with  sufficient  information  to  understand  the  scope  and  costs  of  each  of 
the  major  HF  activities? 

The  reports  provide  some  idea  of  scope,  but  not  in  enough  detail.  Does  not  believe  that  costs 
reflect  the  full  scope  of  the  underlying  activities.  Providing  the  cost  of  each  activity  would  not  be 
particularly  useful. 


Do  the  reports  provide  you  with  sufficient  information  to  imderstand  the  logistical  aspects  and 
timeline  for  implementing  each  of  the  major  HF  activities? 

These  are  generally  well  described,  but  not  convinced  of  the  suggested  sequential  timeline,  e.g.  the 
time  required  for  developing  requirements  from  analysis  is  likely  to  be  much  more  extensive  than 
is  reflected  in  the  reports. 

The  risks  associated  with  not  doing  selected  aspects  of  the  HFE  are  not  easily  comprehended 
except  for  ensuring  better  initial  analysis  and  requirements  definition  early  in  project  development. 
It  would  be  useful  to  provide  more  tangible  examples  of  HFE  work  and  to  show  how  risk  is 
mitigated. 


Do  the  reports  have  sufficient  detail  for  you  to  imderstand  each  of  the  major  HF  activities?  Yes. 


Would  you  like  to  have  seen  more  concrete  military/army  examples  in  some  areas'? 

Yes,  would  like  to  see  one  example  followed  throughout  for  mission  analysis,  function  and  task 
analysis. 


How  would  you  see  the  reports  assisting  you  in  the  requirements  specification  process  as  part  of 
SOR? 

Need  specifically  worded  statements  to  incorporate  HFE  paragraphs?  Descriptions  of  human 
factors  engineering,  mission,  function  and  task  analysis  would  be  useful  in  specifying 
requirements. 
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Are  there  remaining  areas  of  uncertainty  concerning  the  implementation  of  a  HF  program  plan 
that  are  not  addressed  by  the  reports? 

Fusing  HFE  with  iterative  development 


Can  you  provide  a  brief  historical  review  of  the  way  you  considered  implementing  HF  into  the 
initial  project  development  scheme? 

Based  on  an  LFCS  process  model,  the  staff  planning  process  was  a  core  concept  used  as  baseline 
(taught  at  Staff  College).  This  was  developed  into  Chameleon  to  provide  a  demonstration  vehicle 
for  user-oriented  concepts  and  functions.  The  lack  of  task  analysis  was  discovered  later  in  the 
process,  as  was  the  need  to  develop  scenarios  based  upon  mission  analysis.  Later  came  to  the  view 
that  task  analysis  needs  to  be  a  part  of  iterative  prototyping. 

Was  there  any  useful  documentation  available  to  have  helped  your  decision  making  in  this 
process ? 

Auto  Combat  Information  System  (ACIS)  Systems  Requirement  Analysis  (SRA). 


Would  the  availability  of  the  present  reports  have  shaped  your  thinking  differently? 

Yes,  sections  three  and  four  make  a  good  argument  for  the  HFE  case.  This  can  be  combined  with 
the  lessons  learned  from  LFCS  where  they  obtained  data  inefficiently  and  in  a  non-integrated 
manner  by  going  from  function  to  function. 
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A.2  Point  form  transcription  of  the  interview  with  the  TBCS  PD  at  DND  HQ  July 
24  1998. 

Initial  general  comments 

Overall,  the  document  is  dense  and  information  is  hard  to  find  under  time  pressure.  Both 
documents  need  to  be  linked.  The  average  military  officer  would  not  have  time  to  go  through. 
People  rarely  take  time  to  take  a  few  hours  to  read  through  paper  documentation. 

Terminology:  what  is  a  system?  The  physical  hardware/ s oftware  plus  the  process/people/  this  is  a 
CCIS;  the  CCIS  is  only  there  to  support  the  commander.  The  sub-component  is  the  C2IS,  which 
comprises  the  physical  components  of  the  system  used  by  many  different  personnel.  Requirements 
for  CCIS  are  at  a  higher  level,  only  some  of  which  will  involve  the  C2IS.  Deficiencies  will  show 
that  some  require  hardware  solutions,  procedure  solutions,  changes  to  command  structure,  doctrine. 
It  is  all  information  management. 

Reports  focus  more  on  C2IS,  whereas  he  would  like  to  be  able  have  a  report  that  addresses  the 
larger  CCIS  issues. 


Questions  and  responses 

Have  you  had  an  opportunity  to  read  all  of  the  sections  of  the  reports ? 

Has  just  read  the  Guidelines  Document. 

Have  you  had  an  opportunity  to  read  and/or  study  any  similar  documents  previously  -if so-  what 
were  they ? 

On  ARDS  project  read  some  of  the  Engel  and  Townsend  reports,  previously  back  in  1 992. 

Were  the  reports  organised  in  manner  that  made  sense  to  you? 

Yes.  But  there  was  too  much  to  weed  through  in  the  time  available  in  a  typical  DLR  office;  -need 
faster  access  to  information. 


Were  the  reports  organised  in  manner  that  enabled  you  to  understand  the  various  HEPP 
components  and  how  they  related  to  one  another ? 

Yes  -  but  4.3/4. 6  should  go  together.  The  organisation  should  be  comprehensible  for  a  na'ive  reader. 


Were  the  reports  written  in  the  appropriate  level  of  technical  language ?  Yes. 

Do  you  think  that  the  typical  military  officer  would  have  difficulty  in  understanding  the  technical 
aspects  of  the  reports?  If  so,  which? 

Yes.  But  those  with  no  experience  in  HF  and  concepts  would  need  a  special  course  or  learning 
module  on  CD-ROM. 

Was  it  useful  to  divide  the  reports  into  separate  guidelines  and  manual  sections?  Yes 
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On  the  whole  which  do  you  think  army  users  would fine  more  useful? 

Would  need  both;  the  requirements  for  detail  and  the  manual  for  cross  reference. 

Were  there  any  new  concepts  in  the  reports  that  you  had  not  encountered  previously? 

If  so,  what  were  they? 

Yes,  the  training  requirements  for  simulations  (these  are  often  overlooked). 

Were  there  any  concepts  with  which  you  were  previously  familiar,  but  became  clearer  to  you  as  a 
result  of  reading  the  reports? 

The  table  on  development  phases  and  requirements  was  useful. 

Also  the  section  on  MOP’s  and  MOE’s  provides  useful  generic  information.  He  does  not  need 
more  detail  here  because  the  actual  measures  depend  too  much  on  local  context. 

Another  important  section  was  that  on  testing  for  validation-  through  prototypes  and  mock-ups. 

Were  there  concepts  you  would  have  expected  to  find  in  the  reports  but  were  missing? 

Yes,  the  big  system  (CCIS)  view  is  missing. 


Detailed 

Which  aspects  of  the  reports  would  have  been  useful  at  the  very  outset  of  the  TBCS  project? 

How  HF  fits  into  general  systems  engineering.  Also  of  use  is  the  breakdown  of  HF  activities  by 
phase. 

DID’s  would  have  been  useful. 


Would  you  have  done  anything  different  at  the  outset  of  the  project  had  the  reports  been  available? 
Would  the  reports  have  changed  your  approach  to  moving  quickly  to  the  rapid-prototyping 
approach  adopted  by  TBCS  and  DREV? 

Because  of  his  own  background  (a  “systems  approach  type”)  and  his  post-graduate  education  and 
experience  with  ARDS,  the  PD  came  to  the  project  believing  that  HF  analysis  must  drive  system 
development.  But  the  project  was  in  place  already  and  committed  to  Chameleon  development.  He 
tried  to  steer  the  project  in  the  direction  of  having  HF  become  the  core  initial  activity,  since  the 
ongoing  prototyping  was  being  done  out  of  context  of  the  HF  analysis.  The  reports  reinforce  the 
need  to  take  a  system  wide  view.  TBCS  should  focus  on  requirements  development  at  battlefield 
level  and  below  and  then  whether  LFCS  can  meet  that  functionality.  HF  methodology  should  be 
used  to  drive  the  process. 


Do  the  reports  provide  guidance  on  what  are  the  next  most  important  HF  considerations?  How  do 
thry  do  that? 

Yes,  would  provide  good  overall  guidance  for  future  work. 


Do  the  reports  provide  insight  into  the  various  HF  responsibilities  throughout  the  project 
acquisition  cycle?  Yes. 
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If  not,  in  which  of  the  following  areas  do  you  think  you  would  use  it  the  most:  pre-contract, 
requirements  definition,  post  contract-system  development,  implementation  and  fielding? 

The  reports  stress  the  need  for  a  strong  front-end  process,  but  HF  must  run  throughout  the  entire 
development  cycle. 

Would  the  reports  assist  the  project  office  in  planning  and  integration  the  HF  effort  throughout  the 
design,  development,  fielding  cycle?  How  -  strengths  and  weaknesses? 

Yes.  However,  the  reports  would  be  considerably  better  if  the  information  was  easier  to  access. 
The  reports  clearly  show  how  the  project  fits  into  the  overall  system  engineering  cycle. 

Do  the  reports  provide  you  with  sufficient  information  to  understand  the  benefits  of  each  of  the 
major  HF  activities?  —  Yes  in  the  “User  Manual”. 


Do  the  reports  provide  you  with  sufficient  information  to  understand  the  scope  and  costs  of  each  of 
the  major  HF  activities? 

Would  like  to  see  much  more  on  scope  and  costs  to  help  trade-off  issues. 

Do  the  reports  provide  you  with  sufficient  information  to  understand  the  logistical  aspects  and 
timeline  for  implementing  each  of  the  major  HF  activities ? 

Yes,  for  the  most  part  (in  the  “User  Manual”). 


Do  the  reports  have  sufficient  detail  for  you  to  understand  each  of  the  major  HF  activities?  Yes. 

Would  you  like  to  have  seen  more  concrete  military/army  examples  in  some  areas? 

Yes,  need  many  more  of  these  for  the  “User  Manual”. 


How  would  you  see  the  reports  assisting  you  in  the  requirements  specification  process? 

Essentially,  the  most  important  information  provided  by  the  reports  concerns  the  immediate  way  to 
proceed  for  developing  requirements  statements  for  the  next  generation  system. 


Are  there  remaining  areas  of  uncertainty  concerning  the  implementation  of  a  HF  program  plan 
that  are  not  addressed  by  the  report?  No. 


Can  you  do  a  brief  historical  review  of  the  way  you  considered  implementing  HF  into  the  initial 
project  development  scheme ?  How  did  that  change  as  the  project  evolved? 

The  prior  approach  was  disjointed  with  no  coherent  process.  Demonstration  prototypes  of  system 
functionality  drove  the  development  approach.  As  the  project  evolved  a  new  integrated  approach 
developed  with  HF  as  core. 
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Was  there  any  useful  documentation  available  to  have  helped  your  decision  making  in  this 
process? 

Not  really  -  nothing  that  was  easily  accessible  or  made  available  to  requirements  officers. 

Would  the  availability  of  the  present  reports  have  shaped  your  thinking  differently? 

Very  definitely,  although  it  really  only  reinforced  own  beliefs  as  to  what  would  be  the  appropriate 
approach  for  system  development. 
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A.3  Comments  provided  in  writing  by  the  TBCS  PMO  Staff  Officer  in  response  to 
outline  questions  submitted  in  advance  by  HSI. 

GENERAL 

Have  you  had  an  opportunity  to  read  and/or  study  any  similar  documents  previously  -if so  -  what 
were  they? 

Yes,  I  have  read  plenty  of  documents  and  here  are  3  I  have  available  at  work  : 

a.  ACCES  Assessment  of  Command  and  Control  During  a  Division-Level  CPX,  Late  Spring 
1991. 

b.  Cognitive  Analysis,  Design  and  Evaluation  Framework 

c.  Task  Analysis  for  Conduct  Intelligence  Planning 


Were  the  reports  organised  in  manner  that  made  sense  to  you? 

I  had  some  difficulty  but  overall  it  was  easy  enough  to  understand. 


Were  the  reports  organised  in  manner  that  enabled  you  to  understand  the  various  HEPP 
components  and  how  they  related  to  one  another? 

I  had  trouble  understanding  section  5  of  User  Manual:  Guidelines  for  Human  Factors  Engineering 
Requirements  and  why  they  explained  it  the  way  they  did.  Although  the  document  had  some 
impact  analysis  of  inadequate  statement  of  deficiencies  for  example,  it  did  not  address  the  risk 
taken  by  a  Project  PD  if  the  decision  was  made  not  to  do  a  certain  part  of  the  HFE  plan.  Risk 
analysis  in  general,  was  poorly  done. 


Were  the  reports  written  in  the  appropriate  level  of  technical  language? 

Yes,  the  documents  were  written  to  an  appropriate  level  but  you  must  remember  I  have  attended 
many  meetings  and  tried  to  apply  the  process  over  the  last  3  years.  Someone  coming  in  for  the  first 
time  would  find  it  slightly  difficult  to  follow  in  my  opinion. 


Do  you  think  that  the  typical  military  officer  in  charge  of  an  C2IS  acquisition  project  would  have 
difficulty  in  understanding  the  technical  aspects  of  the  reports?  If  so,  which? 

Yes,  I  think  he  would  and  the  HFE  plan  is  a  good  example  of  this.  I  also  feel  a  new  officer  would 
find  it  difficult  understanding  the  overall  process.  Once  a  basic  understanding  is  achieved  of  the 
subject  matter  the  document  starts  to  become  more  important.  I  believe  an  officer  in  charge  of  such 
projects  should  receive  some  basic  courses  on  HFE  such  as  the  ones  available  through  the  "learning 
tree"  program. 

Was  it  useful  to  divide  the  reports  into  separate  guidelines  and  implementation  documents? 

Yes,  I  like  the  divisions. 
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Were  there  any  new  concepts  in  the  reports  that  you  had  not  encountered  previously?  If  so,  what 
were  they? 

Yes,  the  document  makes  several  comparisons  with  civilian  management  information  systems. 
This  helps  in  understanding  and  putting  the  military  environment  in  proper  context.  These  sections 
in  the  document  helped  me  understand  the  importance  of  a  contractor  who  understands  and  has 
built  military  systems. 

Were  there  any  concepts  with  which  you  were  previously  familiar,  but  became  clearer  to  you  as  a 
result  of  reading  the  reports? 

The  contents  of  CCIS  HFE  requirements  documents,  in  particular  DIDs  (pg  106). 


Were  there  concepts  you  would  have  expected  to  find  in  the  reports  but  were  missing? 

Yes,  I  was  hoping  to  see  and  get  a  better  understanding  of  Risk  Analysis. 

DETAILED 

Which  aspects  of  the  reports  would  have  been  useful  at  the  very  outset  of  the  TBCS  project? 

The  document  provides  a  good  outline  and  that  is  what  is  most  important  for  starting  a  project.  The 
document  provides  a  plan  without  which  you  have  nothing. 

How  would  they  have  been  useful? 

An  important  part  of  any  project  is  understanding  the  deficiencies  of  a  system  and  how  one  goes 
about  to  determine  these  deficiencies. 


Would  you  have  done  anything  different  at  the  outset  of  the  project  had  the  reports  been  available? 

I  would  have  looked  at  the  development  of  mission  scenarios  or  a  composite  scenario  to  examine 
the  requirements. 


Would  the  reports  have  changed  your  approach  to  moving  quickly  to  the  rapid-prototyping 
approach  adopted  by  TBCS  and  DREV? 

I  was  never  in  favour  of  the  rapid  prototyping  approach  so  these  reports  would  not  have  changed 
my  mind. 


Given  the  current  stage  of  development  of  the  TBCS  project,  do  the  reports  provide  insight  into 
incorporating  HF  aspects  into  system  design  should  the  project  move  ahead  in  the  foreseeable 
future? 

Yes,  without  question  HF  aspects  would  need  to  be  incorporated  into  the  design  regardless  of  the 
stage  of  development. 
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Does  it  provide  guidance  on  what  are  the  next  most  important  HF  considerations?  How  does  it  do 
that? 

Yes,  it  does  by  giving  the  desk  officer  a  process  to  follow  regardless  in  what  stage  of  development 
the  project  maybe  in. 

Do  the  reports  provide  insight  into  the  various  HF  responsibilities  throughout  the  project 
acquisition  cycle? 

To  some  degree. 

If  not,  in  which  of  the  following  areas  do  you  think  you  would  use  it  the  most:  pre-contract, 
requirements  definition,  post  contract-system  development,  implementation  and fielding? 

I  feel  these  reports  are  helpful  throughout  the  complete  life  cycle  of  a  project.  They  will  for  a  new 
project,  in  the  pre-contract  and  requirements  definition  phase  provide  a  guidance  and  awareness 
into  HFE  that  an  officer  did  not  have  previously.  I  believe  in  this  phase  these  reports  will  be  of 
most  value. 

Would  the  reports  assist  the  project  office  in  planning  and  integration  the  HF  effort  throughout  the 
design,  development,  fielding  cycle?  How  -  strengths  and  weaknesses? 

The  strengths  are  in  the  overall  HFE  process  and  the  reports  that  need  to  be  produced  in  the  forms 
of  DIDs.  The  weakness  is  on  risk  analysis  as  mentioned  in  my  earlier  comments. 

Do  the  reports  provide  you  with  sufficient  information  to  understand  the  benefits  of  each  of  the 
major  HF  activities? 

Yes  to  some  extent;  this  is  an  area  that  could  be  expanded. 


Do  the  reports  provide  you  with  sufficient  information  to  understand  the  scope  and  costs  of  each  of 
the  major  HF  activities? 

No  not  in  any  significant  detail  unless  you  consider  5%  of  the  overall  project  budget.  The  report 
does  bring  out  one  important  detail  and  that  is  the 

Do  the  reports  provide  you  with  sufficient  information  to  understand  the  logistical  aspects  and 
timeline  for  implementing  each  of  the  major  HF  activities? 

No  it  does  not  but  I  would  imagine  this  would  be  extremely  difficult  to  produce. 

Do  the  reports  have  sufficient  detail  for  you  to  understand  each  of  the  major  HF  activities? 

I  do  not  have  sufficient  experience  and  since  I  have  never  developed  requirements  using  this 
process  it  is  difficult  for  me  to  comment  on  the  major  HF  activities. 

Would  you  like  to  have  seen  more  concrete  military/army  examples  in  some  areas? 

I  think  examples  are  an  ideal  way  of  enforcing  the  process.  The  few  examples  that  are  present  in 
the  reports  are  good  as  it  assisted  me  in  understanding  what  was  happening.  Overall  most  definitely 
the  reports  could  use  more  examples. 
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How  would  you  see  the  reports  assisting  you  in  the  requirements  specification  process? 

I  am  not  sure  what  you  mean  here?  Are  you  asking  system  or  software  requirements?  The  reports 
give  me  a  guideline  to  follow  as  I  mentioned  in  my  earlier  comments  for  system  requirements.  I 
still  have  some  doubts  on  how  a  system  engineer  will  deal  with  these  developed  requirements  to 
actually  build  a  system. 


Are  there  remaining  areas  of  uncertainty  concerning  the  implementation  of  a  HF  program  plan 
that  are  not  addressed  by  the  reports? 

No,  none  that  I  haven't  already  mentioned. 


What  are  they?  What  improvements  could  be  made? 

Risk  analysis  and  examples  are  the  two  areas  I  would  like  to  see  improved. 


Can  you  do  a  brief  historical  review  of  the  way  you  considered  implementing  HF  into  the  initial 
project  development  scheme? 

N/A 

How  did  that  change  as  the  project  evolved? 

N/A 

Would  the  availability  of  the  present  reports  have  shaped  your  thinking  differently? 

Yes,  the  availability  of  these  reports  would  have  changed  my  initial  plans.  I  also  had  a  draft  version 
of  these  reports  and  I  used  them  make  many  of  my  decisions. 
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A.4  Note  form  transcription  of  interview  with  the  Project  Manager  ARDS/ADM  at 

DND  HQ  December  11  1998. 


Initial  general  comments 

The  PM  described  himself  as  being  an  end-user  (artillery  officer)  employed  as  deputy  project 
manager  during  the  development  phase  and  has  no  engineering  or  computer  science  background. 
He  assumed  his  present  position  as  the  project  was  being  wound  down.  He  has  been  involved  with 
C2  issues  since  1988. 

He  was  previously  familiar  with  the  MANPRINT  program. 

The  hardest  part  of  any  project  is  trying  to  figure  out  what  problem  it  is  you  are  trying  to  solve.  HF 
will  certainly  facilitate  this  but  it  most  not  be  seen  as  a  end  but  strictly  as  a  means  to  an  end. 


Major  comments 

If  the  reports  had  been  available  at  the  start  of  the  project,  it  would  have  been  of  considerable 
benefit  for  planning  and  prioritising  the  initial  major  HF  tasks  to  be  done. 

The  business  case  for  conducting  a  full  HF  program  needs  to  be  spelled  out.  A  cost/benefit 
analysis  would  be  useful.  More  information  concerning  the  actual  estimates  of  time  and  cost  should 
be  provided  together  with  the  expected  benefits/risks. 

The  major  weakness  with  the  reports  is  that  it  fails  to  show  how  to  integrate  the  various  HF 
activities  with  a  general  systems  engineering  planning  approach.  It  reads  as  if  the  HF  plan  drives 
the  systems  engineering  process,  instead  of  being  integrated  with  it  (particularly  in  areas  such  as' '' 
mission  and  function  analysis). 

The  reports  only  present  one  approach  to  addressing  the  HFE  issues  in  respect  to  CCIS.  They  may 
very  well  be  other  equally  valid  approaches/methods  to  look  at  complex  system  such  as  Quality 
Function  Deployment. 

The  reports  make  a  sound  case  for  the  early  identification  of  deficiencies  and  provide  an  excellent 
account  of  the  process  to  be  followed  for  such  identification. 


Questions  and  responses 

Have  you  had  an  opportunity  to  read  all  of  the  sections  of  the  reports?  YES 

Received  a  copy  earlier  in  year  and  used  it  as  the  basis  for  a  SOW  for  CMC  -  centred  on 
developing  statements  of  deficiency. 

Believed  there  was  a  strong  need  to  document  existing  deficiencies. 

Arrived  at  this  decision  independently,  but  the  validation  provided  by  the  reports  for  this  approach 
provided  a  good  start. 
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Have  you  had  an  opportunity  to  read  and/or  study  any  similar  documents  previously  -  if  so  -  what 
were  they  ? 

Attended  lectures  on  MANPRINT  while  on  Tech  Staff  Course  in  Shrivenham. 

Has  read  general  systems  engineering  books. 

Participated  in  HF  workshop  sponsored  by  PMO  ARDS  ADM  and  held  at  MacDonald  Detwiler 
and  also  received  a  copy  of  the  proceedings. 


Were  the  reports  organised  in  manner  that  made  sense  to  you? 

Yes  —  it  flowed  well.  It  was  easy  to  understand  because  it  follows  the  operational  planning  process 
closely;  therefore  an  army  person  would  relate  to  it. 


Were  the  reports  organised  in  manner  that  enabled  you  to  understand  the  various  HEPP 
components  and  how  they  related  to  one  another? 

Yes.  Although  found  some  difficulty  with  section  5.4.1  -  launching  HFE  plan. 

The  major  weakness  with  the  organisation  is  to  see  how  to  integrate  the  various  sections  with  a 
general  systems  engineering  planning  approach. 

This  relationship  should  be  made  explicit  with  a  schematic  model. 

The  reports  tell  you  what  to  do  but  are  not  clear  on  when  to  do  it  or  how  to  integrate  with  other 
processes. 

Not  immediately  evident  on  where  to  make  HF  trade-offs. 

The  requirements  development  process  needs  to  be  better  tied  to  the  systems  engineering  plan. 


Were  the  reports  written  in  the  appropriate  level  of  technical  language? 
YES  -pitched  at  the  right  level. 


Do  you  think  that  the  typical  military  officer  would  have  difficulty  in  understanding  the  technical 
aspects  of  the  reports?  If  so,  which?  NO 

Was  it  useful  to  divide  the  reports  into  separate  guidelines  and  implementation  sections? 

Yes.  These  would  have  different  uses  depending  on  the  phase  of  the  program 

On  the  whole  which  do  you  think  army  users  would find  more  useful? 

Both  must  be  read  since  they  are  tied  together. 

Were  there  any  new  concepts  in  the  reports  which  you  had  not  encountered  previously?  NO 
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Were  there  any  concepts  with  which  you  were  previously  familiar,  but  became  clearer  to  you  as  a 
result  of  reading  the  reports? 

No,  because  of  previous  good  exposure. 

There  was  a  reverse  problem  (concepts  made  less  clear  in  the  reports):  The  section  on  function 
allocation  is  not  clear,  especially  4A.2.3.  The  relationship  to  Fitt’s  list  could  be  explained.  Page  28 
contains  too  much  technobabble. 

Would  like  to  have  seen  some  information  on  function  allocation  among  personnel. 

Would  have  liked  some  cross-reference  to  other  ways  of  looking  at  problem,  especially  quality 
function  deployment  (QFD).  This  approach  could  result  in  a  different  way  of  doing  HF.  QFD  starts 
with  a  functional  breakdown-  but  focuses  more  quickly  on  fundamental  needs  of  the  users.  Tries  to 
pin  down  one  problem  that  is  critical  very  early.  This  approach  needs  to  be  adapted  from  hardware 
design  to  C2.  There  may  be  a  problem  in  trying  to  map  HFE  approach  to  QFD. 

For  PMs  it  may  be  hard  to  evaluate  HF  content  in  a  contract  proposal  which  uses  a  QFD  approach. 

Good  for  concept  development  but  hard  to  see  how  can  generically  fit  in  with  systems  engineering 
plans. 

Were  there  concepts  you  would  have  expected  to  find  in  the  reports  but  were  missing  NO. 


Detailed  Questions 

Which  aspects  of  the  reports  would  have  been  useful  at  the  very  outset  of  the  ARDS/ADM  project? 

The  section  dealing  with  capability  deficiencies  would  have  been  useful  to  formalise  a 
methodology. 

The  reports  would  make  it  easier  to  assign  priorities  and  make  early  decisions.  Also  forces  you  to 
look  at  the  entire  requirement  for  all  aspects  of  the  system  design  (HSI  note:  presumably  by 
considering  HFE  requirements  along  with  the  standard  systems  engineering  requirements). 

Allows  client  to  do  homework  and  define  the  operational  concept  early.  The  mission/function 
analysis  leads  naturally  to  understanding  the  required  operational  architecture-who  does  what  to 
whom,  when  and  how. 

The  reports  lay  out  a  formal  process.  This  makes  for  generic  consistent  statements  and  the  process 
of  arriving  at  them  is  accountable  and  transparent. 

The  reports  clearly  identify  the  role  of  client  and  level  of  effort  required  in  order  to  properly  define 
requirements. 


Do  the  reports  provide  guidance  on  what  are  the  most  important  HF  considerations?  How  do  they 
do  that? 

Yes  it  will  be  easier  now  to  know  what  to  look  for,  and  how  to  develop  the  systems  engineering 
plan  to  integrate  HF.  Spells  out  need  for  HFE  downstream  .  (As  an  aside,  it  was  noted  that  the 
HF/systems  engineering  team  needs  to  be  cohesive  and  to  demonstrate  clear  expertise.)  More  than 
just  an  aside,  the  Team  (system/HF  engineers)  most  not  only  be  experienced  in  their  respective 
fields  but  must  also  demonstrated  working  experience  together! 
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Do  the  reports  provide  insight  into  the  various  HF  responsibilities  throughout  the  project 
acquisition  cycle? 

Yes  —  but  would  have  liked  to  have  seen  this  done  within  the  context  of  system  engineering  plan. 
As  written,  the  reports  make  it  look  as  if  HF  sits  on  top  of  everything  as  opposed  to  being 
integrated. 


Would  the  reports  assist  the  project  office  in  planning  and  integration  the  HF  effort  throughout  the 
design,  development,  fielding  cycle?  How  -  strengths  and  weaknesses? 

Problems  of  integration  as  described  above.  For  example  tasks  listed  in  section  3.2  a,b,c,  are  all 
found  in  systems  engineering  plans.  Therefore,  how  does  the  approach  recommended  differ  from 
good  systems  engineering? 


Do  the  reports  provide  you  with  sufficient  information  to  understand  the  benefits  of  each  of  the 
major  HF  activities? 

No,  would  like  to  see  a  section  on  the  business  case  analysis. 


Do  the  reports  provide  you  with  sufficient  information  to  understand  the  scope  and  costs  of  each  of 
the  major  HF  activities? 

No,  would  like  to  see  more  detail 


Do  the  reports  provide  you  with  sufficient  information  to  understand  the  logistical  aspects  and 
timeline  for  implementing  each  of  the  major  HF  activities? 

No,  these  are  important  areas  that  are  not  covered  adequately. 

Do  the  reports  have  sufficient  detail  for  you  to  understand  each  of  the  major  HF  activities? 
YES.  But  would  like  more  information  on  how  IV&V  fits  in. 


Would  you  like  to  have  seen  more  concrete  military/army  examples  in  some  areas? 

Not  in  own  case,  but  this  would  possibly  be  more  useful  for  a  requirements  officer. 

Need  checklist  for  what  is  a  good  requirement.  A  few  good  examples  would  go  a  long  way. 

How  would  you  see  the  reports  assisting  you  in  the  requirements  specification  process? 

The  reports  do  not  give  much  guidance  in  moving  from  the  HF  analysis  to  the  statement  of 
requirements. 
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General  systems  requirements  for  function  allocation  are  not  dealt  with  adequately.  Cannot  just 
focus  energies  on  either  the  human  or  the  hardware  side. 


Are  there  remaining  areas  of  uncertainty  concerning  the  implementation  of  a  HF  program  plan 
that  are  not  addressed  by  the  reports? 

Less  obvious  is  how  to  integrate  into  systems  development  cycle. 


Can  you  do  a  brief  historical  review  of  the  way  you  considered  implementing  HF  into  the  initial 
project  development  scheme? 

HF  issues  were  only  mentioned  briefly  initially.  But  a  new  PM  brought  a  strong  HF  perspective  - 
but  it  occurred  too  late. 

Part  of  the  problem  was  developing  R&D  expertise  within  Canada;  this  was  a  more  complex 
process  than  just  buying  a  piece  of  hardware. 

Whole  effort  must  exist  within  a  program  reality  -  especially  if  there  is  no  clear  mission. 


Huma nsystems  Incorporated 


C2IS:  Guidelines  and  User  Manual  Review 
Annex  A 


Page  A- 18 


A.5  Note  form  transcription  of  interview  with  the  HFE  Program 
Manager/Government  Focal  Point  Displays  and  Program  Manager  for  HFE  for  the 
Internal  Communication  System  R/SAOC  Modernisation  Program,  December  11 


1998. 


General  Comments 

This  is  a  useful  document  and  it  would  have  been  highly  desirable  to  have  had  something  similar  in 
place  at  the  start  of  the  project.  It  would  have  guided  much  of  the  early  processes  and  would  have 
spelled  out  clearly  the  need  for  the  contractor  delivering,  and  committing  to,  a  viable  HEPP. 

The  sections  on  developing  statements  of  deficiencies  are  potentially  very  useful,  as  are  the 
statements  on  the  HEPP  deliverables. 

The  section  on  budget  is  useful  but  should  be  enhanced  with  some  detailed  budget  examples  drawn 
from  previous  projects  in  order  to  enhance  the  authority  of  the  document.  These  examples  should 
deal  with  making  the  business  case  for  HF  and  provide  an  indication  of  the  cost  payback  for  the 
investment. 

There  is  a  need  to  develop  a  generic  “briefing  package”  that  could  be  used  to  make  the  case  for  the 
HF  program. 

There  should  be  a  section  on  the  risks/benefits  of  the  various  HF  tasks. 

There  should  be  a  section  outlining  the  need  for  early  identification  of  the  software  required  for  HF 
test  and  evaluation,  since  the  specific  design  for  this  software  will  have  to  be  integrated  into  the 
overall  systems  software  program.  (Note:  a  similar  case  can  be  made  for  the  development  of 
system  maintenance  software). 

The  relationship  between  the  HF  plan  and  other  approaches  to  the  systems  engineering  process 
needs  to  be  explained. 

The  reports  are  not  as  “user  friendly”  as  it  could  be.  It  can  be  quite  verbose,  contains  repeated 
material  and  places  too  many  demands  on  memory  across  the  various  sections. 


Specific  Questions 

Have  you  had  an  opportunity  to  read  all  of  the  sections  of  the  reports? 

Read  all,  but  skimmed  over  some  details  in  the  various  DIDs. 

Have  you  had  an  opportunity  to  read  and/or  study  any  similar  documents  previously  -  if  so  -  what 
were  they? 

NO  -  no  previous  specific  knowledge,  just  that  acquired  as  part  of  an  aerospace  systems  course. 
Gathered  expertise  along  the  way  from  DCIEM,  who  provided  mostly  verbal  guidance. 

Used  some  material  from  the  HSI  report  on  MOPs  and  framework  for  evaluation. 
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Were  the  reports  organised  in  manner  that  made  sense  to  you? 

Really  liked  the  User  Manual,  but  was  less  enthusiastic  about  the  Guidelines  report.  Found  this  too 
verbose  and  there  was  too  much  redundancy  for  things  that  are  said  elsewhere. 

The  reports  make  demands  on  the  reader  to  remember  too  much  material  across  different  sections 
of  the  two  reports. 


Were  the  reports  organised  in  manner  that  enabled  you  to  understand  the  various  HEPP 
components  and  how  they  related  to  one  another?  YES 


Were  the  reports  written  in  the  appropriate  level  of  technical  language? 

Found  the  section  on  Requirements  quite  challenging,  even  with  a  background  in  the  material. 
Need  better  introduction  and  lead  in  to  many  sections 
The  document  reads  like  a  textbook. 


Do  you  think  that  the  typical  military  officer  would  have  difficulty  in  understanding  the  technical 
aspects  of  the  reports?  If  so,  which? 

Unless  it  was  the  officer’s  job  to  read  them,  the  reports  would  be  discarded  and  put  down  quickly. 
This  is  because  the  information  is  not  presented  and  organised  in  a  manner  that  makes  it  user 
friendly. 

Not  enough  examples  were  provided  in  the  User  Manual. 


Was  it  useful  to  divide  the  reports  into  separate  guidelines  and  a  user  manual? 
YES:  each  took  a  slightly  different  perspective. 


On  the  whole  which  do  you  think  air  force  users  would find  more  useful? 

User  Manual  for  general  purposes;  the  Guidelines  document  would  be  more  useful  for  contract 
managers. 

Overall  found  the  User  Manual  much  more  useful. 


Were  there  any  new  concepts  in  the  reports  that  you  had  not  encountered  previously? 

Yes  -  budget  information  was  useful.  Would  like  to  see  concrete  examples  to  support  the  authority 
of  document. 


If  so,  what  were  they? 

The  section  on  evaluation  of  contractor  FIFE  plan  was  good. 
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Were  there  any  concepts  with  which  you  were  previously  familiar,  but  became  clearer  to  you  as  a 
result  of  reading  the  reports? 

YES-evaluation  of  the  contractor.  Developing  and  putting  together  the  HEPP  -  really  liked  part 
about  requirements  deficiencies. 

The  section  on  HEPP  deliverables  was  also  good. 


Were  there  concepts  you  would  have  expected  to  find  in  the  reports  but  were  missing? 

A  short  executive  summary  of  major  points  (found  at  end  of  user  manual)  would  be  useful  to  use  to 
promote  “buy-in”  by  others  to  plan. 

The  case  for  HFE  needs  to  be  made  clearly. 

Material  from  the  reports  could  be  added  to  the  program  manager’s  course  provided  by  the 
department. 


Were  there  any  concepts  with  which  you  were  previously  familiar,  but  became  less  clear  to  you  as 
a  result  of  reading  the  reports? 

YES-the  Guidelines  document  was  not  clear  on  different  test  approaches  (storyboard,  etc).  This 
was  hard  to  map  on  to  own  experience  with  such  methods. 


Detailed 

Which  aspects  of  the  reports  would  have  been  useful  at  the  very  outset  of  the  project? 

Making  the  case  for  the  HF  effort. 

Ensuring  that  the  HEPP  is  a  contractor  deliverable. 

Would  have  resulted  in  a  much  clearer  statement  of  deficiencies  (note:  has  never  been  on  a  project 
that  starts  with  deficiency  requirements. 

Would  strongly  buy  into  this  process. 


How  would  they  have  been  useful? 

Helps  to  provide  detailed  deficiencies  in  a  disciplined  manner. 

Would  you  have  done  anything  different  at  the  outset  of  the  project  had  the  reports  been  available? 

The  major  areas  impacted  would  be  identification  of  deficiencies  and  budget.  The  reports  would 
have  helped  to  justify  budget  and  plan  budget  better. 

Would  have  ensured  deliverable  of  HEPP  from  contractor. 
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Does  it  provide  guidance  on  what  are  the  next  most  important  HF  considerations?  How  does  it  do 
that? 

NO:  not  immediately  obvious  -  need  to  dig  below  surface  to  get  this. 

Do  the  reports  provide  insight  into  the  various  HF  responsibilities  throughout  the  project 
acquisition  cycle? 

Yes  -but  does  not  fit  the  R/SAOC  IPT  (integrated  program  teams)  model  well,  this  is  a  core  part  of 
IPPD  process  that  has  been  adopted  by  US  Department  of  Defence.  The  Canadian  Air  Force  is 
moving  towards  adopting  this  approach  and  the  R/SAOC  program  was  used  as  a  model. 

The  reports  are  weak  on  the  role  of  IV&V. 

The  reports  also  do  not  show  clearly  the  chain  of  command  in  the  HF  team  in  project  management 
hierarchy.  There  is  a  need  to  spell  out  reporting  relationships  and  to  encourage  the  HF  plan 
management  to  be  high  in  the  program  management  hierarchy. 

The  reports  are  very  strong  on  the  pre-contractual  work  that  needs  to  be  done. 

Would  the  reports  assist  the  project  office  in  planning  and  integration  the  HF  effort  throughout  the 
design,  development,  fielding  cycle?  How  -  strengths  and  weaknesses? 

YES  —  it  is  a  big  step  forward  and  provides  clear  guidelines. 

Do  the  reports  provide  you  with  sufficient  information  to  understand  the  benefits  of  each  of  the 
major  HF  activities? 

NO  .  It  would  have  been  useful  to  have  a  section  on  this.  However,  in  reality  it  is  more  probable 
that  a  local  expert  would  make  the  necessary  decisions,  because  trade-offs  have  to  be  done  on  a 
continuing  basis.  Probably  could  not  devise  a  manual  to  deal  with  all  of  the  complexities  that 
change  over  time. 

More  could  be  done  to  bring  our  risks  of  not  doing  the  particular  task. 

Do  the  reports  provide  you  with  sufficient  information  to  understand  the  scope  and  costs  of  each  of 
the  major  HF  activities? 

Yes  in  terms  of  scope,  but  no  in  terms  of  cost  issues.  Would  like  information  on  what  should  be  the 
relative  weighting  of  expenditures.  Also  more  detail  needed  on  level  of  effort. 

Do  the  reports  provide  you  with  sufficient  information  to  understand  the  logistical  aspects  and 
timeline  for  implementing  each  of  the  major  HF  activities? 

Need  more  examples  of  an  overall  business  plan. 
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Do  the  reports  have  sufficient  detail  for  you  to  understand  each  of  the  major  HF  activities? 

YES,  but  need  to  highlight  earlier  in  the  User  Manual  the  need  to  get  HF  expertise  (this  gets  left  to 
end).  Need  to  show  then  the  relationship  between  the  project  office  HF  effort  and  those  of  the 
contractor  and  IV&V  team. 


Would  you  like  to  have  seen  more  concrete  military  examples  in  some  areas? 

YES  -  this  is  a  major  need  -  both  for  comprehension  and  making  the  business  case. 

The  reports  should  emphasise  the  importance  of  creating  a  database  to  track  the  HF  plan  (e.g. 
usability  trial  comments,  IPT  notes,  who  said  what  and  what  was  done).  Suggested  the  adoption  of 
a  usability  tracking  matrix  to  maintain  a  log  of  when  each  issue  is  to  be  addressed  (which  build  and 
evaluation  trial  in  an  iterative  process),  the  priority  of  each  issue  and  the  outcome. 


How  would  you  see  the  reports  assisting  you  in  the  requirements  specification  process? 

By  doing  a  complete  analysis  of  deficiencies. 

Making  a  clear  case  for  HF  to  get  program  manager  “buy-in”. 

Are  there  remaining  areas  of  uncertainty  concerning  the  implementation  of  a  HF  program  plan 
that  are  not  addressed  by  the  reports?  YES. 

What  are  they?  What  improvements  could  be  made? 

Would  like  to  see  a  briefing  package  put  together  which  summarises  the  main  material,  and  that 
would  make  a  clear  case  effectively  to  be  used  in  a  short  briefing. 

There  could  also  be  a  separate  document  “Making  the  case  for  HF”  to  support  any  presentation 
package. 

Would  like  to  see  checklists  provided  to  guide  the  collection  of  data  in  usability  trials. 

The  reports  should  make  it  clear  that  users  of  systems  are  not  just  operators  but  also  maintenance 
personnel. 

Need  to  describe  a  process  for  data  traceability. 

5.1:  the  system  description  should  be  extended  to  include  what  you  would  like  the  system  to  do  in 
specific  terms. 

5.1.6:  Testing  should  be  required,  not  optional  as  implied  here. 

5.2.4:  Section  is  difficult  to  follow 

5.2.5:  c,  d,  could  be  clearer:  need  also  to  identify  those  repetitive  tasks  which  are  candidates  for 
automation. 

Liked  the  sections  on  using  prototypes  and  simulations  for  training  purposes,  although  the  section 
could  be  further  developed  (e.g.  the  early  development  of  symbology  in  the  emerging  system  could 
be  immediately  adapted  for  training  purposes). 

Liked  the  specification  of  the  HF  requirements  up  front  in  the  program  plan. 
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5.3.1:  Approach  is  unrealistic.  Some  compromises  must  be  made. 

Would  like  to  see  the  HF  plan  clearly  state  the  contractual  requirements  in  delivering  a  HEPP. 
Need  a  way  to  ensure  that  the  contract  manager  briefs  other  members  of  the  contractor  team  on  the 
centrality  of  the  HEPP.  The  contractor  needs  to  be  specific  about  the  relationship  between  the 
HEPP  and  the  overall  systems  engineering  plan.  The  HEPP  must  be  seen  to  have  the  same  status  as 
the  software  and  hardware  engineering  plans  and  must  be  integrated  with  these  processes.  Must 
ensure  that  the  contractor  buys  into  this. 

Instead  of  a  general  reference  to  Milspecs,  program  managers  are  expected  to  refer  to  the 
appropriate  sections  for  call-up.  The  reports  does  not  facilitate  this  selection  process. 

User  Manual 

Liked  how  to  use  manual  and  specifically  sections  3.4,  4.2.6.7, 4.2.7,  4.3. 1.2,  4.5,  5.2.2 
4.2.6.5:  this  section  on  MOPs  needs  considerable  strengthening. 

4.3.5:  definition  of  prototype  is  too  constraining 

5.3 .2.6:  Concerned  about  reference  to  schedule  “delay”  here:  this  is  a  red  flag  to  system 
developers.  The  emphasis  should  be  on  the  value  added,  not  the  potential  for  delay. 

5.5:  This  needs  to  be  supplemented  and  developed  and  condensed  into  a  separate  briefing  package 
of  2-3  pages  only. 

Comments  on  experiences  with  HF  analytical  processes  used  on  the  R/SAOC  project 
HF  Analyses 

The  initial  task  analysis  produced  lots  of  data,  but  this  has  not  yet  been  exploited  to  its  full 
potential.  There  was  too  much  data  to  be  useful.  An  independent  assessment  of  the  data  would  be 
useful  to  provide  future  guidance  on  how  it  could  best  be  implemented  into  the  program. 

The  workload  analysis  data  describing  workload  peaks  provided  only  a  limited  insight  into  the 
underlying  deficiencies.  The  data  did  not  address  situation  awareness  or  communication  issues  to 
the  level  required. 

The  list  of  tasks  produced  by  the  task  analysis  was  widely  used  by  the  contractor. 

The  separate  cognitive  task  analysis  is  still  reaping  additional  benefits.  The  CTA  generated  a 
different  type  of  data  from  the  TA  and  was  of  benefit  in  different  areas.  The  TA  was  seen  to 
describe  what  tasks  were  done,  whereas  the  CTA  addressed  how  tasks  were  done  and  the 
information  format  required  to  support  these  tasks. 

The  requirements  development  process  was  able  to  use  the  output  of  the  TA.  This  needed  to  be 
supplemented  with  material  from  the  CTA,  which  provided  information  on  detailed  design  issues 
concerning  the  flow  of  information,  and  the  design  of  specific  screens. 

The  TA  and  OSD  decomposition  helped  to  generate  a  checklist  for  function  development,  but  the 
modelling  data  were  not  used  systematically. 

Issues  need  to  be  addressed  concerning  the  competing  priorities  for  time  and  resources  that  may 
result  from  two  potentially  competing  aspects  of  the  HF  program.  First,  the  generation  and  use  of 
HF  data  to  directly  support  the  development  of  the  specific  program  in  question,  second,  the  need 
for  the  HF  program  to  also  provide  a  more  general  HF  benefit  for  research  into  broader  military 
applications. 
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Annex  B.  Overview  of  the  concept  of  situation 
awareness 


The  term  situation  (sometimes  situational)  awareness  is  frequently  heard  within  the  military, 
human  factors  and  systems  development  communities.  In  many  instances,  military  users  of  the 
phrase  have  adopted  it  to  describe  a  state  or  process  that  is  specific  to  their  own  operational 
circumstances.  System  developers  may  also  use  the  term  in  a  unique  way  to  describe  a  particular 
technology,  for  example,  a  new,  mobile  tactical  display  that  could  be  potentially  deployed  in 
military  vehicles  has  been  described  as  an  "advanced  situation  awareness  system.”  The  widespread 
variation  in  the  usage  of  the  phrase  is  done  without  the  realisation  that  there  is  a  reasonable 
consensus  among  the  HF  community  on  the  meaning  of  the  term. 

This  consensus  has  been  reached  through  a  variety  of  quantitative  and  analytical  studies  over  the 
last  decade,  and  the  interested  reader  is  directed  to  reference  10,  which  provides  a  good  overview 
of  recent  empirical  and  theoretical  work  in  the  field. 

Some  examples  of  the  variation  in  the  way  situation  awareness  has  been  defined  in  the  HF 
literature  are: 

“the  pilot’s  knowledge  of  his  surroundings  in  the  light  of  his  current  goals  ” . 

“situation  awareness  is  the  accessibility  of  a  comprehensive  and  coherent  situation 
representation  which  is  continuously  being  updated  in  accordance  with  the  results  of 
recurrent  situation  assessments  ” 

“situation  awareness  is  up-to-the-minute  comprehension  of  task  relevant  information  that 
enables  appropriate  decision  making  under  stress 

Perhaps  the  most  widely  quoted,  and  possibly  accepted,  definition  is  provided  by  Endsley 
(reference  11):  “ the  perception  of  the  elements  in  the  environment  within  a  volume  of  time  and 
space,  the  comprehension  of  their  meaning  and  the  projection  of  their  status  in  the  near  future 

In  Endsley’s  conception,  situation  awareness  is  to  be  clearly  differentiated  from  both  decision 
making  and  action  (or  performance).  Situation  awareness  serves  as  the  cognitive  antecedent  to  the 
decision  making  process  by  providing  the  appropriate  information  for  decision  input.  Situation 
awareness  is  part  of  the  individual’s  internal  model  of  the  environment.  It  follows,  therefore,  that  if 
situation  awareness  is  degraded,  then  sub-optimum  decision  making  will  likely  occur.  The 
corollary  is  not  true  however,  having  complete  and  perfect  information  as  part  of  the  operator’s 
situation  awareness  is  no  guarantee  that  the  appropriate  or  correct  decision  will  be  made. 

Endsley  has  characterised  three  fundamental  aspects  of  situation  awareness,  which  she  refers  to  as 
“levels”  as  follows. 

Level  1:  Perception  of  the  elements  of  the  environment 

This  involves  the  processes  of  search,  detection,  recognition  and  identification  of  the  relevant 
status,  features  and  attributes  of  the  environment  that  are  pertinent  to  the  goals  in  hand.  The  usual 
assumption  is  that  the  information  required  to  develop  situation  awareness  is  potentially  obtained 
from  multiple  sources  in  the  operating  environment. 
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Level  2:  Comprehension  of  the  current  situation 

Comprehension  is  achieved  through  the  integration  and  synthesis  of  the  relevant,  disparate 
information  that  is  acquired  through  Level  1 .  In  this  process  the  individual  goes  beyond  the 
detection  of  the  information  that  is  present  in  the  situation  and  seeks  to  determine  the  significance 
and  meaning  of  it.  In  some  cases,  this  may  produce  a  holistic  picture,  or  internal  representation,  of 
the  environment.  In  complex  environments  the  ability  to  achieve  this  level  of  comprehension  will 
be  dependent  upon  the  skill  and  knowledge  that  is  acquired  through  experience. 

Level  3:  Projection  of  future  status 

This  refers  to  the  requirement  of  an  individual  to  anticipate  or  envisage  the  future  status  or  actions 
of  the  elements  in  the  environment  over  a  near  time  frame.  Endsley  is  not  specific  about  the  exact 
nature  of  the  time  frame.  This  may  be  contextually  dependent  and  related  to  the  rate  of  information 
that  needs  to  be  attended  to,  and  the  underlying  time  scale  that  is  required  to  achieve  goals.  In  an 
air  combat  context,  the  time  frame  may  be  in  matters  of  seconds,  in  naval  engagements,  it  could  be 
minutes  or  hours. 

These  three  levels  really  describe  processes  relevant  to  the  establishment  and  maintenance  of 
situation  awareness,  rather  than  a  state  of  mind.  However,  for  many  users  of  the  concept,  situation 
awareness  also  means  the  internal  state  arising  from  the  level  2  process.  This  internal  state  is 
frequently  described  in  pictorial  or  graphical  terms  when  military  users  talk  about  “losing  the 
picture”,  “seeing  the  big  picture”  and  “battlefield  visualisation”. 

From  the  perspective  of  HFE  analysis  particularly  in  the  context  of  CCIS,  the  concept  of  situation 
awareness  may  provide  some  useful  perspectives  that  complement  mission,  function  and  workload 
analysis.  Situation  awareness  and  workload  are  intimately  inter-related,  since  if  a  system  fails  to 
support  the  user’s  needs  in  acquiring  and  maintaining  situation  awareness  of  the  operational 
environment,  ensuing  workload  may  be  high.  However,  it  has  been  argued  that  to  some  extent  the 
two  concepts  are  orthogonal.  Thus,  one  can  have  high  workload  and  low  situation  awareness  in 
some  circumstances.  This  could  occur  when  an  individual  has  no  idea  of  what  is  going  on  and 
working  very  hard  to  find  out,  or  if  the  volume  of  information  is  so  great  that  attention  is  too 
narrowly  focussed  on  a  subset  of  information. 

As  an  example,  we  could  apply  an  analysis  based  upon  Endsley’s  three  levels  to  address  the  design 
requirements  for  a  tactical  map.  An  analysis  of  the  operator’s  level  1  needs  would  identify  the 
specific  information  that  must  be  readily  detected  and  comprehended.  This  in  turn  would  have 
design  implications  for  how  the  data  may  be  displayed  to  enhance  conspicuity  and  meaning.  This 
would  be  augmented  with  an  analysis  based  on  level  2  requirements,  which  would  identify 
information  that  must  be  readily  integrated  in  terms  of  the  local  operational  goals. 

If  design  requirements  addressed  to  level  2  needs  are  ignored,  the  resulting  system  may  fail  to 
support  the  user  in  the  difficult  task  of  integrating  information,  for  example,  between  local  and 
more  global  views  of  the  battlefield.  This  in  turn  will  cause  an  increase  in  workload  from  two 
sources.  First,  the  user  may  take  additional  time  (the  primary  driver  of  workload)  to  switch  back 
and  forth  between  map  views,  in  order  to  build  an  integrated  picture.  Second,  the  user  may  be 
forced  to  perform  mental  transformations  of  the  internal  representation  of  the  data  in  order  to 
provide  the  appropriate  battlefield  picture.  The  analysis  of  requirements  for  information 
integration  to  form  a  coherent  and  comprehensible  view  of  the  operating  environment  is 
particularly  important  for  system  designs  in  which  the  user  must  interact  with  multiple  displays  and 
other  information  sources  to  acquire  the  necessary  state  of  situation  awareness. 
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An  analysis  that  takes  into  account  level  3  situation  awareness  requirements  provides  important 
information  on  how  to  design  for  environments  in  which  the  operator  must  project  the  current 
situation  ahead  in  time.  This  projection  can  take  a  variety  of  forms,  e.g.  envisaging  what  the 
disposition  of  forces  in  the  battlefield  will  look  like  at  some  tactically  relevant  future  time, 
planning  a  route,  or  assigning  mental  priorities  for  a  critical  path  for  upcoming  tasks  in  a  multi¬ 
tasking  environment. 

A  common  approach  to  performing  an  analysis  of  situation  awareness  requirements  is  to  start  by 
identifying  the  specific  goal  that  an  operator  is  trying  to  accomplish  at  a  given  time.  This  is 
followed  by  determining  the  critical  pieces  of  information  that  must  be  integrated  into  situation 
awareness  for  the  goal  to  be  successfully  accomplished.  Special  attention  is  given  to  information 
that  must  be  perceived  and  understood  quickly  and  which  information  is  of  less  salience  to  the 
operator.  (Note  -  depending  upon  the  goal  at  hand,  information  that  is  less  salient  for  one  goal, 
may  be  highly  critical  for  another).  The  next  step  is  to  consider  how  the  detected  information  is 
integrated  by  the  user  to  provide  comprehension.  This  may  require  that  the  format  of  disparate 
pieces  of  information  be  transformed  in  a  particular  way  to  facilitate  their  integration.  For  this 
purpose,  it  will  be  desirable  to  establish  the  timeframe  over  which  the  information  will  be  used  and 
the  dynamics  of  any  decision  making  resulting  from  the  current  awareness  of  the  situation.  For 
tasks  involving  planning,  or  anticipation  of  the  way  in  which  the  operational  context  will  change  in 
the  future,  additional  analysis  will  be  required  to  determine  which  aspects  of  information  have  to 
be  projected  forward  over  time  and  how  the  operator  will  use  such  information. 

In  summary,  an  analysis  of  the  operator’s  goals  and  associated  states  of  situation  awareness  has  the 
potential  to  provide  a  significant  value-added  component  to  the  processes  of  requirements 
specification,  and  subsequently  detailed  system  design.  This  is  achieved  by  taking  into 
consideration  how  the  information  provided  by  a  CCIS  must  be  matched  to  the  particular  way  in 
which  information  will  be  used  by  the  operator  in  a  specific  operational  context.  A  successful 
analysis  will  therefore  contribute  to  improving  the  probability  that  new  system  designs  will  contain 
the  necessary  flexibility  and  adaptability  to  meet  a  user’s  varying  information  needs  contingent 
upon  the  current  goals  in  hand. 
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